Today I was wrestling with Azure Data Factory to get a list of folders; filter out an unwanted folder, and then iterate over these folders to do some magic. Getting the list of all folders gave me some headache, so I want to share my findings to do it right the first time.

1. Create a dataset
2. Get a folder list
3. Add a filter component

Create the dataset

Create a new dataset

Choose type binary

Do not create a dataset with a file type of CSV or one of the other filetypes. This will give you a list of the files.

Enter the container name; in my case “costmanagement”.

Get folder list

Add a Get metadata component

Add the “Child Items” in the field list; nothing else necessary.

When you run now; you can see the output of the metadata component filled with a list of all folders within the container.

Filter unwanted

Enter the filter settings:

Items = @activity(‘GetListofBlobs’).output.childItems

Filter out all folders starting with an underscore: @not(startswith(item().name,’_’))

Conclusion

I’ve tried the same with a datset of CSV and still don’t understand why I get the content of one file. Let me know when you have an explanation. But the above binary dataset works for me.