Rectangle 27 23

A directory is like a file that contains a list of the filenames in it (indeed, you may have accidentally opened a directory in an editor, where you can see it is a mostly-readable list of its contents) -- so it is modified when a file in that directory is added, removed, or renamed.

@Flimm It seems to only update the containing directory's mtime otherwise it would bubble up to the root.

Out of interest, which editors allow you to open a directory directly? Most (even ed and cat) seem to complain, or offer a list of file to open instead.

@Ether: It doesn't "open a directory" as in actually reading the directory's inode's data (or what ever equivalent on other file systems). It just provides a file browser for that directory.

linux - Directory last modified date - Stack Overflow

linux unix ftp
Rectangle 27 3

Afterwards, once the file is displayed. In the display function - printf - Copy the filenames in the array. I guess there is a restriction on the filename size.So that can be the maximum size of the array. In case, you want to save the memory, then you can use realloc and can create the exact number of characters array.

This is a shortcut way to get the data.

#include <dirent.h>
#include <stdio.h>

char name[256][256];

int main(void)
{
  DIR           *d;
  struct dirent *dir;
  int count = 0;
  int index = 0;
  d = opendir(".");
  if (d)
  {
    while ((dir = readdir(d)) != NULL)
    {
      printf("%s\n", dir->d_name);
      strcpy(name[count],dir->d_name);
      count++;
    }

    closedir(d);
  }

  while( count > 0 )
  {
      printf("The directory list is %s\r\n",name[index]);
      index++;
      count--;
  }

  return(0);
}

Although, I have eaten up the memory but it will work for you.

count

get list of file names and store them in array on linux using C - Stac...

c linux list filenames glob
Rectangle 27 4

The problem is not related to awk, but to the shell (how you pass the filenames):

Unquoted command substitution $( ls ) will expand to a list of filenames, but the filenames are subject to word-splitting, so that filenames with embedded spaces are each broken into multiple arguments passed to awk.

This results in awk seeing either nonexistent filenames (at which point a fatal error occurs) or accidentally processing different files (multiple times); e.g., if files file one, file and one all exist in the current directory, awk will not process file one, and instead process both file and one twice.

A simple glob (*) will do in this case, whose expansion results are not subject to work-splitting, and is generally preferable to parsing ls output:

awk 'BEGIN {for ( i=1;i<ARGC;i++ )print "ARGV " i ": [" ARGV[i] "]" }
     FNR==1 {print "FILENAME " ++a ": [" FILENAME "]" }
    ' *

Using an unquoted command substitution to expand to multiple arguments passed to a command (command $(...)) is an anti-pattern in general

$ touch file 'file 1'
$ bash -s - $(ls file 'file 1') <<<'echo "$# args passed: [$1] [$2] [$3]"'
3 args passed: [file] [file] [1]

Note how, even though file 1 was passed with quotes, the target command (an ad-hoc bash script) sees 3 arguments, as a result of the shell having broken file 1 into separate arguments file and 1 (word-splitting), due to unquoted use of $(...) (command substitution). (Note that "$(...)" wouldn't have helped, because the command output is then invariably passed as a single argument.)

The following simplified command causes awk to fail fundamentally, because instead of seeing single filename File One, it sees filenames File and One, neither of which exist:

$ rm -f File One; echo 'hi from File One' > 'File One'
$ awk '{ print FILENAME }' $(ls 'File One')
awk: fatal: cannot open file `File' for reading (No such file or directory)

The above is GNU awk's error message; BSD Awk and Mawk fundamentally behave the same, except for variations in the wording of the error message. All these implementations set the exit code to 2 in this scenario.

if you want to print a space in between words, just say print a, b --> print "FILENAME", ++a, ...

understand but why FILENAME is thus correct in this case (with a spaced name). If it is purely shell expansion, FILENAME will received several bad file names ?

@fedorqui problem is not the print, I use it to show the problem when inside the code. I use something like Files[ ARGV[i]]++ in BEGIN section for catching empty files. this is where my problem appear

@NeronLeVelu It is simply impossible for awk or any other command to recognize an unquoted file name with spaces in it as a single argument. It sounds like you might have a question worth posting as a new question on the site.

forget my last comment. I agree your info, i try just to find a way to recreate a shell interprestation after the $() occur to reseparate the argument at shell levell before awk catch them

awk shell parameter FILENAME vs ARGV - Stack Overflow

shell awk
Rectangle 27 6

Spaces in filenames

If you use f.el, a convenient file and directory manipulation library, you only need function f-entries.

However, if you don't want to use this library for some reason and you are ok for a non-portable *nix solution, you can use ls command.

(defun my-directory-files (d)
  (let* ((path (file-name-as-directory (expand-file-name d)))
         (command (concat "ls -A1d " path "*")))
    (split-string (shell-command-to-string command) "\n" t)))

The code above suffice, but for explanation read further.

man ls
-A, --almost-all
          do not list implied . and ..

With split-string that splits a string by whitespace, we can parse ls output:

The problem is that some filenames may contain spaces. split-string by default splits by regex in variable split-string-default-separators, which is "[ \f\t\n\r\v]+".

-1     list one file per line

-1 allows to delimit files by newline, to pass "\n" as a sole separator. You can wrap this in a function and use it with arbitrary directory.

(split-string (shell-command-to-string "ls -A1") "\n")

But what if you want to recursively dive into subdirectories, returning files for future use? If you just change directory and issue ls, you'll get filenames without paths, so Emacs wouldn't know where this files are located. One solution is to make ls always return absolute paths. According to man ls:

-d, --directory
          list directory entries instead of contents, and do not dereference symbolic links

If you pass absolute path to directory with a wildcard and -d option, then you'll get a list of absolute paths of immediate files and subdirectories, according to How can I list files with their absolute path in linux?. For explanation on path construction see In Elisp, how to get path string with slash properly inserted?.

(let ((path (file-name-as-directory (expand-file-name d))))
  (split-srting (shell-command-to-string (concat "ls -A1d " path "*")) "\n"))

Unix commands have to add a trailing whitespace to output, so that prompt is on the new line. Otherwise instead of:

user@host$ ls
somefile.txt
user@host$

there would be:

user@host$ ls
somefile.txtuser@host$

When you pass custom separators to split-string, it treats this newline as a line on its own. In general, this allows to correctly parse CSV files, where an empty line may be valid data. But with ls we end up with a null-string, that should be omitted by passing t as a third parameter to split-string.

Emacs lisp: `directory-files` - Stack Overflow

file emacs directory elisp interactive
Rectangle 27 12

If you don't want to copy the whole tree (with subdirs etc), use or glob.glob("path/to/dir/*.*") to get a list of all the filenames, loop over the list and use shutil.copy to copy each file.

for filename in glob.glob(os.path.join(source_dir, '*.*')):
    shutil.copy(filename, dest_dir)

Note: You might have to check the glob results with os.path.isfile() to be sure they are filenames. See also GreenMatt's answer. While glob does return only the filename like os.listdir, it still returns directory names as well. The '.' pattern might be enough, as long as you don't have extensionless filenames, or dots in directory names.

copy multiple files in python - Stack Overflow

python file copy
Rectangle 27 25

An unquoted asterisk * will be interpreted as a pattern (glob) by the shell.The shell will use it in pathname expansion. It will then generate a list of filenames that match the pattern. A simple asterisk will match all filenames in the PWD (present working directory).A more complex pattern as */ will match all filenames that end in /. Thus, all directories. That is why the command:

echo */
echo ./*/              ### avoid misinterpreting filenames like "-e dir"

will be expanded (by the shell) to echo all directories in the PWD.

To test this: Create a directory (mkdir) named like test-dir, and cd into it:

mkdir test-dir; cd test-dir
mkdir {cs,files,masters,draft,static}   # safe directories.
mkdir {*,-,--,-v\ var,-h,-n,dir\ with\ spaces}  # some a bit less secure.
touch -- 'file with spaces' '-a' '-l' 'filename'    # and some files:

The command echo ./*/ will remain reliable even with odd named files:

./--/ ./-/ ./*/ ./cs/ ./dir with spaces/ ./draft/ ./files/ ./-h/
./masters/ ./-n/ ./static/ ./-v var/

But the spaces in filenames make reading a bit confusing.

If instead of echo, we use ls, the shell is still what is expanding the list of filenames. The shell is the reason to get a list of directories in the PWD. The -d option to ls makes it list the present directory entry instead of the contents of each directory (as presented by default).

ls -d */

However, this command is (somewhat) less reliable. It will fail with the odd named files listed above. It will choke with several names. You need to erase one by one till you find the ones with problems.

The GNU ls will accept the "end of options" (--) key.

ls -d ./*/                     ### more reliable BSD ls
ls -d -- */                    ### more reliable GNU ls
$ printf "%s\n" */        ### Correct even with "-", spaces or newlines.

And, even better, we could remove the trailing /:

$ set -- */; printf "%s\n" "${@%/}"        ### Correct with spaces and newlines.
$ for i in $(ls -d */); do echo ${i%%/}; done
  • some names (ls -d */) as already shown above.
  • will be affected by the value of IFS.
IFS
  • each newline in the name will start a new echo command.

Finally, using the argument list inside a function will not affect the arguments list of the present running shell. Simply:

$ listdirs(){ set -- */; printf "%s\n" "${@%/}"; }
$ listdirs
--
-
*
cs
dir with spaces
draft
files
-h
masters
-n
static
-v var

This options are safe with several types of odd filenames.

I think you should replace the word Safe with Reliable. Your post makes me think that the other solutions are 'unsafe' (vulnerable or exploitable.)

@AmadoMartinez Done a general s/safe/reliable/. Better?

directory - Listing only directories using ls in bash: An examination ...

bash directory ls
Rectangle 27 23

An unquoted asterisk * will be interpreted as a pattern (glob) by the shell.The shell will use it in pathname expansion. It will then generate a list of filenames that match the pattern. A simple asterisk will match all filenames in the PWD (present working directory).A more complex pattern as */ will match all filenames that end in /. Thus, all directories. That is why the command:

echo */
echo ./*/              ### avoid misinterpreting filenames like "-e dir"

will be expanded (by the shell) to echo all directories in the PWD.

To test this: Create a directory (mkdir) named like test-dir, and cd into it:

mkdir test-dir; cd test-dir
mkdir {cs,files,masters,draft,static}   # safe directories.
mkdir {*,-,--,-v\ var,-h,-n,dir\ with\ spaces}  # some a bit less secure.
touch -- 'file with spaces' '-a' '-l' 'filename'    # and some files:

The command echo ./*/ will remain reliable even with odd named files:

./--/ ./-/ ./*/ ./cs/ ./dir with spaces/ ./draft/ ./files/ ./-h/
./masters/ ./-n/ ./static/ ./-v var/

But the spaces in filenames make reading a bit confusing.

If instead of echo, we use ls, the shell is still what is expanding the list of filenames. The shell is the reason to get a list of directories in the PWD. The -d option to ls makes it list the present directory entry instead of the contents of each directory (as presented by default).

ls -d */

However, this command is (somewhat) less reliable. It will fail with the odd named files listed above. It will choke with several names. You need to erase one by one till you find the ones with problems.

The GNU ls will accept the "end of options" (--) key.

ls -d ./*/                     ### more reliable BSD ls
ls -d -- */                    ### more reliable GNU ls
$ printf "%s\n" */        ### Correct even with "-", spaces or newlines.

And, even better, we could remove the trailing /:

$ set -- */; printf "%s\n" "${@%/}"        ### Correct with spaces and newlines.
$ for i in $(ls -d */); do echo ${i%%/}; done
  • some names (ls -d */) as already shown above.
  • will be affected by the value of IFS.
IFS
  • each newline in the name will start a new echo command.

Finally, using the argument list inside a function will not affect the arguments list of the present running shell. Simply:

$ listdirs(){ set -- */; printf "%s\n" "${@%/}"; }
$ listdirs
--
-
*
cs
dir with spaces
draft
files
-h
masters
-n
static
-v var

This options are safe with several types of odd filenames.

I think you should replace the word Safe with Reliable. Your post makes me think that the other solutions are 'unsafe' (vulnerable or exploitable.)

@AmadoMartinez Done a general s/safe/reliable/. Better?

directory - Listing only directories using ls in bash: An examination ...

bash directory ls
Rectangle 27 1

According to the man page ls lists information about the filenames specified. The shell (bash, sh, zsh etc) expands the * to a list of filenames, so the command being execute is

ls -lR filename1 filename2 filename3 ...

If one of those filenames is a directory then ls will list it recursively.

ls -lR filename

If that filename is the name of a directory, ls will list the contents of the directory recusively, otherwise it will give you details of the file.

ls -lR | grep filename

Or use find as you say

unix - ls -lR * lists all the files in current and subdirectories , bu...

unix ls
Rectangle 27 2

Assuming you don't have filenames with embedded newlines, you can get a list like this:

list=($(find . -name '*.txt'))
printf '%s\n' "${list[@]}"

As other people have pointed out, whether this is useful depends on the context.

@TobySpeight Sure, disclaimer added. In 40 years of working with computers I never saw a single case of actual filename with embedded newlines, but I adding disclaimers about that every other post surely helps keeping hope alive. :)

bash - How to loop through file names returned by find? - Stack Overfl...

bash find
Rectangle 27 2

Assuming you don't have filenames with embedded newlines, you can get a list like this:

list=($(find . -name '*.txt'))
printf '%s\n' "${list[@]}"

As other people have pointed out, whether this is useful depends on the context.

@TobySpeight Sure, disclaimer added. In 40 years of working with computers I never saw a single case of actual filename with embedded newlines, but I adding disclaimers about that every other post surely helps keeping hope alive. :)

bash - How to loop through file names returned by find? - Stack Overfl...

bash find
Rectangle 27 102

Getting Full File Paths From a Directory and All Its Subdirectories

import os

def get_filepaths(directory):
    """
    This function will generate the file names in a directory 
    tree by walking the tree either top-down or bottom-up. For each 
    directory in the tree rooted at directory top (including top itself), 
    it yields a 3-tuple (dirpath, dirnames, filenames).
    """
    file_paths = []  # List which will store all of the full filepaths.

    # Walk the tree.
    for root, directories, files in os.walk(directory):
        for filename in files:
            # Join the two strings in order to form the full filepath.
            filepath = os.path.join(root, filename)
            file_paths.append(filepath)  # Add it to the list.

    return file_paths  # Self-explanatory.

# Run the above function and store its results in a variable.   
full_file_paths = get_filepaths("/Users/johnny/Desktop/TEST")
  • The path I provided in the above function contained 3 files two of them in the root directory, and another in a subfolder called "SUBFOLDER." You can now do things like:
print full_file_paths
['/Users/johnny/Desktop/TEST/file1.txt', '/Users/johnny/Desktop/TEST/file2.txt', '/Users/johnny/Desktop/TEST/SUBFOLDER/file3.dat']

If you'd like, you can open and read the contents, or focus only on files with the extension ".dat" like in the code below:

for f in full_file_paths:
  if f.endswith(".dat"):
    print f
/Users/johnny/Desktop/TEST/SUBFOLDER/file3.dat

python - How do I list all files of a directory? - Stack Overflow

python directory
Rectangle 27 102

Getting Full File Paths From a Directory and All Its Subdirectories

import os

def get_filepaths(directory):
    """
    This function will generate the file names in a directory 
    tree by walking the tree either top-down or bottom-up. For each 
    directory in the tree rooted at directory top (including top itself), 
    it yields a 3-tuple (dirpath, dirnames, filenames).
    """
    file_paths = []  # List which will store all of the full filepaths.

    # Walk the tree.
    for root, directories, files in os.walk(directory):
        for filename in files:
            # Join the two strings in order to form the full filepath.
            filepath = os.path.join(root, filename)
            file_paths.append(filepath)  # Add it to the list.

    return file_paths  # Self-explanatory.

# Run the above function and store its results in a variable.   
full_file_paths = get_filepaths("/Users/johnny/Desktop/TEST")
  • The path I provided in the above function contained 3 files two of them in the root directory, and another in a subfolder called "SUBFOLDER." You can now do things like:
print full_file_paths
['/Users/johnny/Desktop/TEST/file1.txt', '/Users/johnny/Desktop/TEST/file2.txt', '/Users/johnny/Desktop/TEST/SUBFOLDER/file3.dat']

If you'd like, you can open and read the contents, or focus only on files with the extension ".dat" like in the code below:

for f in full_file_paths:
  if f.endswith(".dat"):
    print f
/Users/johnny/Desktop/TEST/SUBFOLDER/file3.dat

python - How do I list all files of a directory? - Stack Overflow

python directory
Rectangle 27 102

Getting Full File Paths From a Directory and All Its Subdirectories

import os

def get_filepaths(directory):
    """
    This function will generate the file names in a directory 
    tree by walking the tree either top-down or bottom-up. For each 
    directory in the tree rooted at directory top (including top itself), 
    it yields a 3-tuple (dirpath, dirnames, filenames).
    """
    file_paths = []  # List which will store all of the full filepaths.

    # Walk the tree.
    for root, directories, files in os.walk(directory):
        for filename in files:
            # Join the two strings in order to form the full filepath.
            filepath = os.path.join(root, filename)
            file_paths.append(filepath)  # Add it to the list.

    return file_paths  # Self-explanatory.

# Run the above function and store its results in a variable.   
full_file_paths = get_filepaths("/Users/johnny/Desktop/TEST")
  • The path I provided in the above function contained 3 files two of them in the root directory, and another in a subfolder called "SUBFOLDER." You can now do things like:
print full_file_paths
['/Users/johnny/Desktop/TEST/file1.txt', '/Users/johnny/Desktop/TEST/file2.txt', '/Users/johnny/Desktop/TEST/SUBFOLDER/file3.dat']

If you'd like, you can open and read the contents, or focus only on files with the extension ".dat" like in the code below:

for f in full_file_paths:
  if f.endswith(".dat"):
    print f
/Users/johnny/Desktop/TEST/SUBFOLDER/file3.dat

python - How do I list all files of a directory? - Stack Overflow

python directory
Rectangle 27 5

Accepts a list of filenames via [2:], does no exception handling:

#!/usr/bin/env python
import re, sys, os

for f in filter(os.path.isfile, sys.argv[2:]):
    for line in open(f).readlines():
        if re.match(sys.argv[1], line):
            print line

sys.argv[1] resp sys.argv[2:] works, if you run it as an standalone executable, meaning

chmod +x

what's the difference between re.match and re.search ?

search
match

regex - Grep and Python - Stack Overflow

python regex grep
Rectangle 27 3

The most harsh way to check if a file would be a valid filename on you target OSes is to check it against a list of properly tested filenames.

valid = myfilename in ['this_is_valid_name.jpg']

Expanding on that, you could define a set of characters that you know are allowed in filenames on every platform :

valid = set(valid_char_sequence).issuperset(myfilename)

But this is not going to be enough, as some OSes have reserved filenames.

AFAIK, Python does not offer such helpers, because it's Easier to Ask Forgiveness than Permission. There's a lot of different possible combinations of OSes/filesystems, it's easier to react appropriately when the os raises an exception than to check for a safe filename domain for all of them.

Please use a set in your first approach :)

python - Check if a filename is valid - Stack Overflow

python
Rectangle 27 16

Automatic variables are set by make after a rule is matched. They provide access to elements from the target and prerequisite lists so you dont have to explicitly specify any filenames. They are very useful for avoiding code duplication, but are critical when defining more general pattern rules.

There are seven core automatic variables:

In addition, each of the above variables has two variants for compatibility with other makes. One variant returns only the directory portion of the value. This is indicated by appending a D to the symbol, $(@D), $(<D), etc. The other variant returns only the file portion of the value. This is indicated by appending an F to the symbol, $(@F), $(<F), etc. Note that these variant names are more than one character long and so must be enclosed in parentheses. GNU make provides a more readable alternative with the dir and notdir functions.

What do the makefile symbols $@ and $< mean? - Stack Overflow

makefile
Rectangle 27 1

As of 18/04/2017, no simple black or white list of characters and filenames is evident among the answers to this topic - and there are many replies.

The best suggestion I could come up with was to let the user name the file however he likes. Using an error handler when the application tries to save the file, catch any exceptions, assume the filename is to blame (obviously after making sure the save path was ok as well), and prompt the user for a new file name. For best results, place this checking procedure within a loop that continues until either the user gets it right or gives up. Worked best for me (at least in VBA).

Your answer @FCastro is correct from the technical point of view. However from the UX perspective it's a nightmare - the user is forced to play the "type something and I'll tell you if you succeed" game again and again. I'd rather see a message (warning style) telling the user that they have entered an illegal character which will later be converted.

What characters are forbidden in Windows and Linux directory names? - ...

windows linux directory zip filenames
Rectangle 27 5

import csv
import itertools as IT

filenames = ['1.csv', '2.csv']
handles = [open(filename, 'rb') for filename in filenames]    
readers = [csv.reader(f, delimiter=',') for f in handles]

with  open('combined.csv', 'wb') as h:
    writer = csv.writer(h, delimiter=',', lineterminator='\n', )
    for rows in IT.izip_longest(*readers, fillvalue=['']*2):
        combined_row = []
        for row in rows:
            row = row[:2] # select the columns you want
            if len(row) == 2:
                combined_row.extend(row)
            else:
                combined.extend(['']*2)
        writer.writerow(combined_row)

for f in handles:
    f.close()
for rows in IT.izip_longest(*readers, fillvalue=['']*2):
In [1]: import itertools as IT

In [2]: readers = [(1,2,3), ('a','b','c','d'), (10,20,30,40)]

In [3]: list(IT.izip_longest(readers[0], readers[1], readers[2]))
Out[3]: [(1, 'a', 10), (2, 'b', 20), (3, 'c', 30), (None, 'd', 40)]

As you can see, IT.izip_longest behaves very much like zip, except that it does not stop until the longest iterable is consumed. It fills in missing items with None by default.

Now what happens if there were more than 3 items in readers? We would want to write

list(IT.izip_longest(readers[0], readers[1], readers[2], ...))

but that's laborious and if we did not know len(readers) in advance, we wouldn't even be able to replace the ellipsis (...) with something explicit.

In [4]: list(IT.izip_longest(*readers))
Out[4]: [(1, 'a', 10), (2, 'b', 20), (3, 'c', 30), (None, 'd', 40)]

Notice the result Out[4] is identical to the result Out[3].

The *readers tells Python to unpack the items in readers and send them along as individual arguments to IT.izip_longest. This is how Python allows us to send an arbitrary number of arguments to a function.

thanks for the solution. I accepted yours because I understand the code best. However, would you mind explaining what "for rows in IT.izip_longest(*readers, fillvalue=['']*2):" does? Especially, the "*readers" is confusing to me...

thanks for the explanation! that was great

python - Combine columns from several CSV files into a single file - S...

python csv
Rectangle 27 1

You should use filenames instead of filename when you provide a list of files:

CITIES_FILES = {
    'city': {
       'filename': 'CO.zip',
       'urls':     ['http://download.geonames.org/export/dump/'+'{filename}']
    },
}

Man, I thought that was OK and I didn't pay attention to it. It works now, thanks!

python - Django Cities import error - Stack Overflow

python django importerror geodjango