Split a very large file in to chunks

Introduction

Many a times we are having to deal with the very large files, be it the logs from the server like catalina.out or any other huge sql files or csv or text files.

These are some artifacts that could have been generated in the server or might contain the data. There might not have been detailed logging systems that can do a RollingFile like appender that can get the files into smaller chunks for us to use easily. In some cases these could be large data which is hard to split into multiple chunks in an export mechanism.

Problem Statement

When we try to open them in an editor like Notepad / Notepad++ / VS Code, we normally run into memory issues and the application gets frozen.

At times, in case of windows, we might have to go to Task Manager , find the application and then End Task so that we can use the other apps or to allow that particular application to be usage

Solution

In this post, we explore on using git bash a tool. We normally use a git bash command line interface that can be used to perform the git operations. In this case we can use a useful command in git bash that can help us here. Open git bash from the windows or linux system.

Assuming that we have a large file to name as largefile.ext which we can think of as having more than 1 GB of size and that we wanted it to be split into smaller chunks of 300mb. The below command can be used from the command line

split largefile.ext -b 300m

Once run, we can use a dir incase of a windows OS or a ls (or) ls -al on a linux system that can help us view the list of files that were created in the process of splitting.

Remarks

I have been trying to literally open server logs by downloading them from server and given that they were very huge, I had to try with different editors and even if the file was opened, a search could take minutes or move the application (editor) into a frozen state which can take mins to unfreeze or might have to end task and restart.

This solution literally saved a lot of my time. Hope it does for the reader as well

Comments

User Authentication schemes in a Multi-Tenant SaaS Application

User Authentication in Multi-Tenant SaaS Apps Introduction We will cover few scenarios that we can follow to perform the user authentication in a Multi-Tenant SaaS application. Scenario 1 - Global Users Authentication with Tenancy and Tenant forwarding In this scheme, we have the SaaS Provider Authentication gateway that takes care of Authentication of the users by performing the following steps Tenant Identification User Authentication User Authorization Forwarding the user to the tenant application / tenant pages in the SaaS App This demands that the SaaS provider authentication gateway be a scalable microservice that can take care of the load across all tenants. The database partitioning (horizontal or other means) is left upto the SaaS provider Service. Scenario 2 - Global Tenant Identification and User Authentication forwarding In the above scenario, the tenant identification happens on part of the SaaS provider Tenant Identification gateway. Post which, the

Handling exceptions in the Executor service threads in Java

Introduction This is a continuation post on the exception handling strategies in the threads in Java. For Introduction, please read this post The second post is available here This post addresses the problem statement "How to use the exception handlers in the threads spawned by the Executor Service in Java?" Not all times, we will be using Thread classes to run our threads because we have to manage a lot of the underlying logic for managing threads. There is ExecutorService in Java which comes to the rescue for the above problem. In the previous posts, we have discussed on how to handle the exceptions in plain threads. However, when using executor service, we do not create / manage threads, so how do we handle exception in this case. We have a ThreadFactory as an argument which can be used to customize the way threads are created for use within the ExecutorService . The below snippet of code leverages this feature to illustrate the exception handling, wherein we create a

Download CSV file using JavaScript fetch API

Downloading a CSV File from an API Using JavaScript Fetch API: A Step-by-Step Guide Introduction: Downloading files from an API is a common task in web development. This article walks you through the process of downloading a CSV file from an API using the Fetch API in JavaScript. We'll cover the basics of making API requests and handling file downloads, complete with a sample code snippet. Prerequisites: Ensure you have a basic understanding of JavaScript and web APIs. No additional libraries are required for this tutorial. Step 1: Creating the HTML Structure: Start by creating a simple HTML structure that includes a button to initiate the file download. <!DOCTYPE html> < html lang = "en" > < head > < meta charset = "UTF-8" > < meta name = "viewport" content = "width=device-width, initial-scale=1.0" > < title > CSV File Download </ title > </ head > < body >

SFTP and File Upload in SFTP using C# and Tamir. SShSharp

The right choice of SFTP Server for Windows OS Follow the following steps, 1. Download the server version from here . The application is here 2. Provide the Username, password and root path, i.e. the ftp destination. 3. The screen shot is given below for reference. 4. Now download the CoreFTP client from this link 5. The client settings will be as in this screen shot: 6. Now the code to upload files via SFTP will be as follows. //ip of the local machine and the username and password along with the file to be uploaded via SFTP. FileUploadUsingSftp("172.24.120.87", "ftpserveruser", "123456", @"D:\", @"Web.config"); private static void FileUploadUsingSftp(string FtpAddress, string FtpUserName, string FtpPassword, string FilePath, string FileName) { Sftp sftp = null; try { // Create instance for Sftp to upload given files using given credentials sf

Implementing Row Level Security [RLS] for a Multi-Tenant SaaS Application

Row Level Security The need for row level security stems from the demand for fine-grained security to the data. As the applications are generating vast amounts of data by the day. Application developers are in need of making sure that the data is accessible to the right audience based on the right access level settings. Even today, whenever an application was built, the application development team used to spend a lot of time researching the approach, implementing multiple tables multiple logics 25 queries to add filters to manage the data security for every query that gets transferred from the end user request to the application database. This approach requires a lot of thought process, testing and security review because the queries needs to be intercepted, updated and the data retrieval to be validated to make sure the end-users see only the data that they are entitled to. Implementation With the advent of of row level security feature being rolled out in main d

Software And Architecture

Search This Blog