Idea dump (sorry)

[krazyjakee]

• ctrl + a should select all files and folders currently visible
• the job queue should have a priority. I can't view thumbnails in my current view because there is a bunch of jobs, I don't understand what they're doing but they take AGES. If i'm viewing a folder, move the thumbnail job to the top of the queue.

[HeavenVolkoff]

Hi @krazyjakee

• Shortcuts are not fully implemented yet (sorry), we are currently restructuring how they work and there is already an internal issue for that.
• The idea of job prioritization is interesting. I’m not entirely sure if we already have that in place or how difficult it would be to implement. Maybe @fogodev can weigh in, as he’s more familiar with that area than I am.

[krazyjakee]

Thanks so much.

[fogodev]

We do have some sort of prioritization in place already. To scan a location we have 3 Jobs: Indexer, File Identifier and Media Processor. Each one depends on the results of the previous one. Indexer will scan your file system and build the file tree in our database. File identifier will read partially each file to generate a unique identifier for each, namely a cas_id, aside from also identifying the file kind. Media Processor will filter only media files and extract data from them, like exif data from images and ffmpeg metadata from audio and videos. Also generating thumbnails for images and videos.

Each of these jobs are broken down into smaller tasks, these tasks are dispatched to a task system we have internally, which is a queuing system with multiple workers, each task can have priority or not. If a priority task reaches a worker, it will interrupt the processing of the current task if it is a non-priority one and start the priority task ASAP. Aside from this prioritization, the task system processes tasks in FIFO order. Each of the 3 jobs I mentioned already dispatches priority tasks for the root directory and non-priority for any descendants directories and files.

Aside from these jobs, we already have "shallow versions" of these jobs, which doesn't use the job system, but also dispatches tasks. They only operate in the current open directory in the explorer, it doesn't go into any children directory. Tasks emitted by the shallow jobs are always with priority.

To avoid conflicts, we only allow dispatch of shallow scan of a directory if there is no scan jobs running in that location to begin with.

From what you explained, it seems to me that you were trying to scan many locations at same time and opened one of them. It's possible that the jobs for the location you opened were further down the tasks queue. As you had a scan pending for that location, you couldn't run the shallow jobs for the directory you opened.

I have some ideas in mind on how to further increase the responsiveness of the tasks system. But I currently have some other priorities that I need to handle first.

[fogodev]

TL;DR: A job prioritization queuing system is in place with three job types: Indexer, File Identifier, and Media Processor, which depend on each other to scan, identify, and process media files. "Shallow versions" of these jobs exist for the currently open directory. However, they can't run if there are already pending jobs for that directory, which appears to be the issue @krazyjakee is facing. As his tasks system queues seems to be pretty full and tasks are processed in FIFO ordering.