loops in linux

loops in Linux shell scripts

Posted on | by root

Loops in Linux shell scripts is something you will inevitably face once writing more complex scenarios. They are nothing new for those who have at least some sort of programming experience, since they exist in pretty much any programming language. Loops are control structures to repeatedly execute a block of code as long as a specified condition is true. They are essential for automating repetitive tasks and processing large amounts of data efficiently. Since loops are a fundamental programming concept, they of course exist in bash (or any other shell).

Continue reading →
regex negate

negate something in regex using negative lookahead

Posted on | by root

If you wish to negate something in regex, one of the easiest and most reliable ways to do it is to employ negative lookahead assertion. Having a need to split the set of objects into different categories with some “other” / “default” / “the rest” option is one of the most common use cases where this approach can be used.

Continue reading →
wildcard expansion

convert wildcard in file name into relevant static file name

Posted on | by root

Sometimes you might encounter cases when files are named somewhat weird and makes it hard to use them for automated analysis purposes. In such cases, wildcard expansion becomes a very useful concept. For example, some applications tend to write their logs into log files with some additional information in file names, like:

[root@linux ~]# ls -l /var/log/demo/
total 76
-rw-r--r-- 1 root root 19125 Mar 15 13:48 2024-03-13.demo.log
-rw-r--r-- 1 root root 29580 Mar 15 13:49 2024-03-14.demo.log
-rw-r--r-- 1 root root 22100 Mar 15 13:49 2024-03-15.demo.log
[root@linux ~]#

Say log file is being written to as long as calendar date matches the current one. And what if you want to set some automated analysis of such log files, but to do it only for the relevant one, to which application currently writes?

Continue reading →
bash check permissions

check ownership and permissions for absolute path easily

Posted on | by root

I couldn’t count times when asking someone to check permissions in Linux for some file I was given answer which was something like: 

[root@linux ~]# ls -l /var/log/httpd/access_log
-rw-r--r-- 1 root root 17245040 Mar 13 13:38 /var/log/httpd/access_log
[root@linux ~]#

So all looks good – file is readable. But is it really?

Continue reading →
IP to country

retrieving country codes from IP automatically

Posted on | by root | 1 Comment on retrieving country codes from IP automatically

I once had a need to monitor the origin of HTTP requests, track it in real time from web server log. Since this need was actually nothing more but pure curiosity, I was not willing to pay for it. So I started looking for free solutions for this purpose – convert IP to country. Here is what it all ended at…

Continue reading →
read log file in portions

read log files in portions by bytes

Posted on | by root | 2 Comments on read log files in portions by bytes

You may want to read log files for newly appended data. You could do it by knowing how many lines you read before and how many lines you have “now”. But when you have large log files, this approach will become very slow. Try to run “wc -l” on file which has tens of millions of lines and you’ll see how slowly it counts. And you would have to do something like that each time you are reading new portion of data. What to do? Read log in bytes instead. Here is how.

Continue reading →