changelog

• version 621

  • more ratings

  • thanks to a user, we have many new rating options--
  • `options->media viewer` and `options->thumbnails` now let you alter the size of ratings!
  • there are now many new rating shapes to choose from under `services->manage services`: triangles pointing up/down/left/right; diamond; rhombus pointing right/left; pentagon; hexagon and small hexagon; six point star; eight point starburst; hourglass; large + and X shapes
  • also, in an experiment, you can now also set a custom svg to your rating. there's a new folder, `install_dir/static/star_shapes`, that you can drop svgs into to have them available. I've put a couple of simple examples in already. try to do a squarish shape with a clear sillhouette on transparency. we will keep working on this, but feel free to try it out yourself and give feedback. I suspect we'll migrate all the existing custom polygons to nicer svgs in future if and when we nail nice svg border colours and stuff
  • I added a help button to the `manage services` edit panel here, linking to the ratings help, which has a new section on the svg rating shapes
  • I reworked some stuff behind the scenes to more neatly integrate this rating tech and allow for svg ratings. the 'edit service star shape' UI in `manage services` has two dropdowns now, too
  • we are experimenting with fractional pixel sizing and drawing here, so some of the new ratings may seem a little blurry in places and/or not line up well, including between the underlying canvas and the top-right hover pop-up. I'm going to work on this a bit and see what I can do to make it more pixel-perfect
  • we may figure out unicode character ratings in future also

  • misc

  • fixed a bug in the duplicates system's 'pixel hash' test. my current pixel hash is dimensionless, and thus, by accident, a completely flat-colour (or careful checkerboard etc..) image of 100x500 has the same pixel hash as the same colour of 200x250. it obviously doesn't come up much, but I regret the error. rather than recompute everyone's pixel hashes with dimension data, we are first patching the test code to double-check the images both have the same width. it seems ok here, but let me know if your 'must (not) be pixel duplicates' duplicate file searches are now interminably slow
  • split the prefetch settings in `options->speed and memory` into their own box, and updated the labels a bit
  • added a setting to this new box to govern how many pairs the duplicate filter will prefetch. previously this was hardcoded to 1, now it defaults to 5 and you can set whatever
  • when the client or server checks if it is already running, the 'client_running' file's 'process creation time' is now only tested to within a second's accuracy. if you have a bespoke backup workflow that involves creating your own client_running file, it should get caught better now despite float imprecision
  • updated the various Linux install/running-from-source stuff to talk about `libmpv2` as well as `libmpv1`.
  • I rolled out a v620a hotfix last week after my refactoring accidentally broke the manual duplicate filter. sorry for the trouble! I have improved my testing regime to catch this in future

  • duplicates auto-resolution

  • you can now add 'system:known url' and 'system:number of urls' predicates to the 'test A or B' comparators
  • you can now add 'system:number of urls' to the 'test A against B using file info' comparators
  • added unit tests for these, and cleaned up some of the unit tests in this area

  • avif

  • the `pillow-avif-plugin` test last week seemed to go well. this library is now folded into the main builds, and AVIF rendering should be more reliable for the time being (issue #1720)
  • if you run from source, you might like to rebuild your venv this week to get this

  • psd_tools replacement

  • `psd-tools` is no longer used by the program
  • thanks to a user report, we discovered that the Linux build was including an old and vulnerable version of a bz2 decoder. this thing was hanging around because of a drawing library included by a psd parser we use. I am not sure if the decoder was ever called, given what we use the library for. I did a test build of Linux with the bad file simply deleted, and it seemed to boot ok, but I wasn't sure if the file was bundled into a pyd on the Windows and macOS side, and I'm pretty sure it would be included in a source install, so I wasn't really happy
  • ultimately I decided I didn't like having this complicated psd library hanging around when we only needed to pull dimensions, icc profile existence, and pulling an embedded render preview file, so I cobbled together some other backup solutions we already mostly figured out, including an ffmpeg renderer, and wrote a simple file parser and replaced all the psd_tools stuff with our own calls
  • there are some psd files ffmpeg seems not to render as good as psd_tools did, but I'm fine with it for now
  • I seem to be detecting icc profiles that psd tools did not recognise, so I'm scheduling all psds for a 'has an icc profile?' check on update. maybe I'm false-positiving, but I'll make sure the existing store reflects current code at least
  • I also fixed some psd rendering for files that had weird 100% transparency
  • along the way I improved some PSD error messages and handling. PSDs that have no embedded preview should stop spamming that info to the log on thumbnail regen

  • swfrender removal

  • since cleaning was on my mind, I'm removing the swfrender exes too. new swf files you import will just get default 'flash' thumbnails from now on. having these ancient executables hanging around to load unknown flash files is not an excellent idea these days, even though it has been fun
  • the actively developed 'Ruffle' flash emulator seems to be working on a CLI executable that can render flash frames, which could be a much better and more reliable replacement in future
  • I haven't played with it properly yet, but ffmpeg can apparently thumbnail some flv-embed-style flash files, so I'll explore this too

  • refactoring and cleanup

  • moved some ffmpeg stuff from `HydrusVideoHandling` to the new `HydrusFFMPEG`
  • merged all 'ffmpeg was missing' and 'ffmpeg output no content' error handling to single calls and improved coverage
  • wrote some 'render to pipe' stuff for ffmpeg to make some of the ffmpeg based rendering work a little quicker and without the temp dir
  • deleted the `HydrusPSDTools` file
  • moved some more old network reporting mode code to a cleaner unified method

• version 620

  • user gui improvements

  • thanks to a user, we have some more UI features--
  • the options dialog now remembers its last page (can uncheck this if you like under `gui` page)
  • a checkbox under `gui` that says whether to save media viewer size and position on close (normally it only saves on move/resize, but if you regularly use multiple viewers, you may wish to override so a final close saves what you want)
  • the media viewer gets a 'drag' button in the top hover. drag the button, you drag the window. useful if you are using it in the newer frameless mode
  • if you right-click this new button, there are some neat new commands to change the fit the window size to the current mediia size too
  • the new 'resize frame to media' commands are mappable on the 'media viewer' shortcut set. you can set the specific zoom
  • we now have a command for 'zoom to x%', also! same deal, it is now in the 'media viewer' shortcut set
  • the media viewer should be better about saving its position when moved by programmatic window position-setters like Windows Snap
  • two checkboxes under `gui pages` let you promote 'all my files' or 'all local files' buttons to the top of the page picker (for, e.g. if you have many file domains that spill over what the dialog can show)

  • misc

  • wrote an animated webp frame duration parser and integrated it into our file metadata stuff. animated webp files are no longer fixed to 12fps and support variable framerate (issue #1694)
  • on update, all animated webps will be scheduled for a metadata regen to get corrected total duration times
  • fixed some url unit tests I accidentally broke last week because of the defunct url classes I removed, and updated some of the 'how to make url classes and parsers' and client api help to use more generic url examples
  • improved the grammar and general presentation of the 'files being parsed' message text in the 'import files' dialog. this text also tooltips itself in case it gets crazy long
  • the 'quality info' button in the 'edit subscription' dialog is now aware of which queries actually exist on disk versus those that were generated in this 'edit subscriptions' session. it now presents info only on selected queries that actually exist, and if only new queries are selected, it disables itself
  • the system tray checkboxes are now set to false along with being disabled if the current system does not support a system tray (e.g. Docker). the calls these settings make have an additional protection layer that checks if the current system has a system tray (issue #1569)
  • the 'show pending and petitioned groups' and 'show whole chains' checkboxes in the manage tag siblings and parents dialogs now disable if you hit 'show all pairs'. these are inherently true with 'show all pairs'
  • the file maintenance routine that attempts to re-queue known urls for files that are missing or damaged now double-checks that any 'Post URLs' are currently parseable. (some 'decorative' urls have 'Post URL' url classes so as to appear in the media viewer but aren't actually linked to anything)
  • in the 'thumbnails' shortcut set, you can now set a 'select: not selected' command, to invert the current selection
  • the display names of normal pages are now clipped to 256 characters and are better about removing accidentally included newline characters
  • the macOS dialog double-positioning thing we added in March that fixed dialogs slowly creeping like 26 pixels down on every dialog session is now careful not to apply to the main window on boot. it seems the fix was making the main gui move ~100px sideways, no idea why

  • future build and more AVIF fun

  • it looks like the AVIF fix last week was not reliable--some boots it would work, others it would not. I don't know if this is some random fail triggered by the deprecated status, but whatever: we should promptly move to the designated 'use this in the interim while Pillow proper figure it out their end' solution to get good AVIF rendering back (issue #1720)
  • thus I am making another 'future build' this week. I had been planning one for a while, particularly to try out a new mpv dll, but I'm just going to keep it nice and simple this week to test out the AVIF fix. if you are an advanced user, please try it out on your platform and let me know if there are any problems. if you run from source, you can rebuild your venv, and if you select (a)dvanced, you'll get a question if you want to install the AVIF test library
  • assuming no problems, I'll fold this into v621
  • on boot, hydrus now imports the `pillow-avif-plugin` library in preference to the old `pillow_heif` solution
  • as a side thing, it looks like Pillow are going to try slender AVIF binaries in their wheels for 11.3, so this all may get simpler soon
  • also, hydrus now reports 'avif ok' and 'heif ok' separately in various errors and the `help->about` dialog

  • json parsing

  • the json formula now supports two new parsing rules--
  • first, a 'walk back up ancestors' parsing rule. it moves `n` steps back up the parsing stack, so if you have an Object where you need to test for one key's existence but you actually need to grab a sibling or cousin value, you can now walk back and it should work
  • second, a 'filter strings/numbers/bools with string match' parsing rule. if you have grabbed strings or other simple variables, you can test them against a String Match. if you combine this with the 'walk back' step, you can now test Object values and then walk back up and grab a different thing
  • wrote some unit tests for the existing and new json formula rules

  • duplicates auto-resolution

  • fixed an issue with renaming existing rules. due to a saving bug, renaming rules was causing duplicate entries behind the scenes. you might get an update message about it--if you do, your new rule may have rolled back to a previous version. I will have paused it, to be safe, so if you were hit by this, please double-check your rule is named as you wish and the settings are all good and then set it back to semi-automatic or automatic if you are happy
  • added a maintenance job to the cog to clear the cached pair counts that the rules use to talk about their progress. if there's ever a miscount, this will fix it
  • when you approve/deny some pending pairs, the selection now tries to preserve to the earliest selection you had before. previously it always fell back to position 0
  • stopped the duplicates auto-resolution work from sending unintended 'I am not idle' signals as it did file search, prohibiting idle mode from kicking in
  • fixed a stupid typo error when you say to add a new comparator rule but cancel the 'select which type' add dialog
  • fixed an issue where a duplicates auto-resolution table wasn't being deleted correctly on rule delete. not a big deal, just a cleanup thing

  • duplicates auto-resolution exact match detection

  • _tl;dr: I think I can do the 'A is a resize/re-encode of B' in future_
  • I did some research and bashed my head against different strategies, and I think I have figured out the core of a routine that can differentiate between resize/re-encode 'exact match' duplicates and files that have significant changes such as corrections or watermarks or recolours. It uses a bunch of math to break the images into normalised tiles, compute a weighted 'wasserstein distance' (earth-mover distance) of the tiles' Lab channel histograms, and reviews those scores, and their mean, variance, and skewness to exclude various classes of differing files
  • I still have a bit of work to actually plug it in, since each judgment here requires a full image render for both in the pair and some of the UI isn't yet ready to handle a ~1 second delay per pair. I also need to figure out a nicer tuning/testing regime to ensure I didn't just overfit for the examples I generated. I think I'll plug it into the manual duplicate filter's comparison statements and we'll see with human eyes how it does. overall, I feel really good about it. I thought this would be a nightmare, but it looks doable. if you want to check my math and send in your own thoughts, check out the new `ClientImageHistograms.py`

  • boring refactoring and cleanup

  • pulled the rich maintenance manager code out of `ClientFiles` to a new `ClientFilesMaintenance`
  • pulled the file storage manager out of `ClientFiles` to a new `ClientFilesManager`
  • pulled the phash code out of `ClientImageHandling` and moved it to a new `ClientImagePerceptualHashes`
  • moved the above files and `ClientFilesPhysical` to new `hydrus.client.files(.images)` modules
  • reworked the images, image tiles, and thumbnails caches to be explicitly named variables in the client controller, mostly to improve some type checking stuff
  • standardised some variable names around the cache access, and made the images and image tiles caches work in media results rather than media objects
  • removed the afaik defunct `cloudscraper` from the code and requirements and a weird insert we needed in the client.spec build templates

• version 619

  • cleaner tags

  • updated the tag filter to exclude many weird unicode characters. all sorts of Control Characters, right-to-left formatting, zero-width spaces, surrogates, and more is now all removed. Zero-Width characters ZWNJ and ZWJ are allowed unless the rest of the tag is only in extended-latin. the hangul filler character is allowed if the tag includes other hangul syllable or jamo. there is no perfect solution here, but a bunch of mis-parses and other garbage is cleaned up with this (issue #1709)
  • **your client will clean its tags on update. this will take several minutes if you sync with the PTR**. you will probably see tens of thousands of bad tags--don't worry about it too much, and everything is logged if you are super interested in the saga of `ç¿«∪ç¸`-legacy-decode-jank, `( ͡° ͜ʖ ͡°)`-that-includes-a-hidden-Byte-Order-Mark, and a relatively common `normaltag[ZWNJ]` that's probably either an IME input mistake or some HTML parsing error from years ago
  • I've been fairly aggressive here, so if I have broken something you do want to input, or something that requires temporary invalid status via an IME, let me know. rest assured, however, that everyone's favourite compound emojis such as `👨‍👩‍👧‍👦` should still work

  • linux build

  • the Linux build failed last week because I missed notifications about the Ubuntu 20.04 runner being retired. sorry for the trouble! this has happened before, so I am going to keep a better check on retiring runner news in future
  • the Linux build is now generated on the 22.04 runner. there _are_ some .so file differences, but it seems to all boot ok, and in one case it actually fixed a previously broken mpv load. also, a normal extract-update seems to have worked in our tests, so we don't think a clean install is strictly needed. You might like to do a clean install anyway, just to be neat: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs

  • misc

  • AVIF rendering is fixed. I confidently wrote some code last week that said 'when Pillow updates to 1.21.1, don't load the recently deprecated AVIF support from our external plugin any more because Pillow will now have it natively', and then the Pillow update happened last week and they decided not to bundle in AVIF in their convenient wheel because it bloated their files. I have undone my version check, frozen the plugin at its current version to keep support, and will check this manually on the actual version of Pillow in future before I switch over again. sorry for the trouble! (issue #1714)
  • added a new `Tell original search page to select exit media when closing the media viewer` checkbox to `options->media viewer` (default on). this lets you turn off the behaviour where your exit media is selected in the thumbgrid when you exit a media viewer (issue #1712)
  • the default value for `When maintenance physically deletes files, wait this many ms between each delete:` in `files and trash` is now 600ms, up from 250ms. if you are set to the old default of 250, the update will bump you up. furthermore, the widget in `files and trash` is now a rich time delta widget rather than just a ms integer spinner
  • the 'are you sure you want to exit the program, these pages say they are not done' yes/no dialog now spawns with yes disabled for 1.2 seconds, just like the archive/delete filter confirmation, enough to jolt you out of an autopilot enter press

  • default downloaders

  • the safebooru parser is more careful about fetching valid associable source urls. it was previously juxtaposing the safebooru domain with non-https-having garbage
  • added a thread parser for holotower
  • deleted some parsers and url classes for long-defunct sites

  • sidecar sorting

  • sidecar objects no longer do a hardcoded sort of their strings before the export step. if you set a different sort via the 'processing' step's string processor, that's directly what will export to the destination
  • all new sidecar objects now start with a--and all existing sidecar objects will update to get a new--processing step that does 'human text sort (asc)'. thus, all sidecars _should_ continue behaving pretty much as they were before, but if you don't want that, you can now edit it!

  • actual vs ideal tags

  • the right-click tag menu that shows current parents and siblings now shows the _ideal_ tag display space, rather than the _actual_. the difference between these two is the ideal is what your settings say whereas the actual is what the client currently has calculated as per _tags->sibling/parent sync->review current sync_. this guy was previously showing the actual calc, which was revealing confusing interim technical states after the preferences changed. I am not sure if this change is correct or helpful and suspect I'll need some better UI around here to quickly detect and explain a discrepancy
  • updated the `/add_tags/get_siblings_and_parents` help to discuss that it fetches actual rather than ideal tags

  • duplicates auto-resolution

  • **the duplicates auto-resolution UI is all enabled**, and I've un-hidden the 'add rule' button. have fun with this new tech, but don't go crazy yet. I think pixel-duplicate pairs are now easy to solve if you have firm preferences about keeping exif etc..
  • finished off a comparison rule for duplicates auto-resolution that tests things like 'A has more than 2x the num pixels as B'. it can test size, width, height, num_pixels, duration, and num_frames and supports equals, not equals, greater than, less than, approx equal (percentage or absolute), and you can set a coefficient (A has more than 2x filesize of B) and/or an absolute delta (A has more than 200 px more height than B)
  • added two hardcoded comparison rules for 'A and B have the same/differing filetypes'. nice and simple way to ensure you are or aren't comparing like to like in a rule
  • the one-file comparators can now do exif, icc profile, and human-readable metadata tests
  • the 'add suggested rules' button now has three choices--the original pixel-duplicates jpeg/png one, and `pixel-perfect pairs - keep EXIF or ICC data` (eliminate pixel duplicate pairs of the same filetype where only one has Exif/ICC data) and `pixel-perfect pairs - eliminate bloat`, (eliminate pixel duplicate pairs of the same filetype where neither have EXIF/ICC data but one is smaller than the other)
  • the 'action' column in the duplicates auto-resolution preview panels now tooltips to the full text. if this becomes like 100 tags and several URLs, you can now read it!
  • although we can now poke at the easiest dupe pairs, I think we are still missing a puzzle piece to differentiate alternates from duplicates programatically, even on 'exact-match' searches. we either need cleverer and higher-resolution phashes or a comparison rule that does hardcoded pixel inspection and allows for 'A is > 99.7% pixel-similar to B' in some semantically rich way so we can automatically differentiate jpeg artifacts from banners or watermarks or even colour-only costume changes. while better and perhaps colour-sensitive phashes may come one day, I am going to go for this pixel comparison tech. this will add render CPU cost to each pair decision, which is going to add some bumps to this whole workflow, particularly the preview window, but I suspected we'd have to do this so I've mostly built for it

  • boring duplicates auto-resolution

  • wrote unit tests for the new system predicate media-result-extraction system for the types it can currently extract
  • wrote unit tests for the new relative pair file info comparator
  • wrote unit tests for the new hardcoded filetype comparator
  • wrote unit tests for the new one-file comparator capabilities
  • the number test widget now emits value-changed signals
  • fixed an issue where many system pred stubs, until now only appearing temporarily, would not serialise
  • fixed an issue with the duplicates auto-resolution preview panel where the 'pass' list was not ordering correct for rules beyond the pixel dupe test rule
  • fixed an issue where the duplicates auto-resolution dialog would not allow you to add more than one new rule per dialog session
  • brushed up the auto-resolution help a bit

  • boring cleanup

  • moved the 'human text sort' stuff from `HydrusData` to `HydrusText`, and improved its sort reliability when strings differ on order of int/str
  • merged the generic tag sort code into the human text sort system, since they were both doing the same thing. cleaned up some bad old ideas along the way
  • fixed some bad old tag processing when generating thumbnail banners and the file has more than two tags of a particular namespace and at least one tag includes non-number data
  • a bunch of places that have some text beside a widget, e.g. the 'show whole chains' checkbox in 'manage tag parents', now copy the tooltip from the widget itself to the text

• version 618

  • misc

  • fixed the tag import options setup on 'force metadata refresh'. sorry, I broke this by accident last week!
  • the weird 'always show system:everything, even if you have more than 10k files' (default off) checkbox under the 'file search' options panel is now replaced with a simpler 'show system:everything' (default on). once users get more experienced, they can turn this off themselves when they notice it. existing users will upgrade to get an appropriate value--if you currently see system:everything, it should default to True to you
  • the crazy 'hide inbox and archive system predicates if either has no files' option, also under 'file search', is removed. this produces more confusion than value
  • gave my top-level help and getting started index pages a pass, making it plainer and more concise for new users and clearing out some unneeded waffling/cringe and defunct or advanced info (the tumblr contact link still had an 'rss' alternate link, lol)
  • fixed an issue where the simple downloader would be unable to resume a parsing job that never finished in a previous session (e.g. if you had network traffic paused). in this case, the gallery result would sit unresolved in the gallery log; the job is now only removed from the pending jobs queue, and the gallery log entry only saved to the log, once the job is complete
  • if the user has the 'archived file delete lock' on, file maintenance jobs that check for missing/incorrect files now recognise if a record to be deleted is delete-locked and now just send the file to the trash instead. you get a different popup letting you know what happened, and the log maps out what happened too if you need to do sophisticated recovery. it sounds backwards to deny the deletion of a record for a file that does not actually exist, but I will not budge on the delete lock--trying to write exceptions is only going to tie us in knots (issue #1706)
  • when repairing the client file storage system, or when doing 'move files now', if the source folder has been stored in the file system as an absolute path despite being beneath the db folder, the system now recognises this and recovers. if neither portable nor absolute path worked (i.e. some very odd path normalisation has happened), the system now stops what it was doing and raises an appropriate error. previously, it was possible to get your client into a situation where it would have doubled-up entries for certain prefixes. this whole system could do with a revamp, I think, especially when I get back to background file migration
  • reworked the last check, last file, and next check column sorts in manage subscription(s) dialogs. the various 'not initialised yet' values were sorting as very old rather than imminent

  • ipfs update

  • I have done a pass over the IPFS service in hydrus, catching it up to what their modern daemon's API wants. if you use a new version of IPFS, hydrus should be able to pin directories again. if you use an old version of IPFS, update it please
  • most importantly, I have removed the `nocopy` feature. the way we did the symlink redirect trick is not supposed to work on modern IPFS. this was always crazy and experimental, so for KISS reasons I will no longer support it
  • I have also removed the native IPFS multihash download from the client. there were a couple of really obscure ways to launch this rickety old download process that could even spawn a tree selection dialog for picking which files in a directory you wanted to get. it was a big mess and a threading nightmare. all IPFS daemons offer web access to multihashes, so if we want this tech back, I think I want to make a downloader and/or URL Class or something instead, and maybe add like 'parse multihash' content update routine, so we can use all the nice downloader UI instead. you can also, of course, just paste an IPFS direct file URL straight into an url downloader and it should work
  • the 'pin new directory' command is fixed!! (issue #1710)
  • if you update your IPFS daemon a lot (as I did this week), I am uncertain if existing pins or directory pins will parse correct, and doubly so if there was nocopy stuff going on before. if nothing is working your end and you can't unpin-then-repin to fix it, I think the best solution is just to prep your currently pinned files in some search pages so you can find them again and then remove/re-add the ipfs service and basically tell hydrus to start over
  • I gave the ipfs help a full pass, here: https://hydrusnetwork.github.io/hydrus/ipfs.html

  • boring cleanup

  • I replaced the last three hacky old duct-taped 'queue' listboxes (where you have a list of texts with up/delete/down buttons beside) with my integrated class that has some more bells and whistles, for instance reactive buttons that only enable when they can fire and selection preservation when you do up/down. the replaced widgets are: the one on the simple downloader page; the list of html parsing formula rules; the list of json parsing formula rules. no more pain in the neck as you reorder parse rules!!
  • the 'queue' listbox now accepts delete key presses
  • the 'queue listbox that can edit now only edits the top selected item
  • the 'queue' listbox has some finer Qt signals working behind the scenes, too
  • deleted some ancient and terrible data fetching and list manipulation functions that were originally from wx and are thankfully now no longer used
  • fixed up some more layout flag transitions for my expand/collapse boxes. the IPFS shares box now starts off collapsed but expands to eat up space

• version 617

  • misc

  • the 'file relationship' commands for 'dissolve duplicate group', 'dissolve alternates group', 'delete false positives', 'reset potential search', and 'remove potential pairs' are now added as mappable actions for the 'media' shortcut set. careful with these--they are advanced and powerful. all of them are wrapped in yes/no confirmation dialogs, every time
  • disabled a 512MB/768MB backing bitmap limit on the `pillow_heif` library we use to render HEIF and AVIF. this allows for loading very large files (issue #1697)
  • improved error handling during an out-of-memory problem during image conversion

  • default downloaders

  • the new e621 parser now grabs the 'director' namespace from e6ai, parsing it as 'creator'. it looks like in many cases this will be 'creator:anonymous director', which I considered filtering out, but this appears to be a conscious 'creator is staying Anon' choice on e6ai's part rather than a 'tagme' style invitation for more work, so it has useful value for our purposes

  • default downloader options

  • tl;dr upshot for today: if you have custom tag/note import options for e621, hit up `network->downloaders->manage default import options` and ensure you have options set for both 'eXXX file page api' and particularly the new 'eXXX gallery page api' entries. use the copy/paste buttons to make it easy
  • the new e621 downloader raised an interesting issue--when a gallery parser produces direct file URLs, there is no way to set custom default tag/note import options for this import. neither file nor gallery urls can have a set default options entry, so it was defaulting to the 'file post' default settings every time--that's ok, but not what we might actually want. I've fixed this today
  • hydrus is now smarter about how it figures out which tag and note import options to use for a particular network download. if the main URL we are going for has no default tag/note import options entry, we now always consult any referral URL as a secondary. previously, the way watcher downloads set up their options was through a hardcoded hack that swapped in the 'parent' url to check; now we check both in all cases and let the first good hit tell us what to do. this will, fingers crossed, fix a bunch of other unusual tag-adding behaviour on downloaders that have post urls that produce separate direct file import objects
  • the default importer options panel now allows gallery url classes! extra text also explains how the system works and how you should engage it. I kind of hate it, but it works. I also don't want to make this even more complicated by enabling File URLs (that would filter and shape the metadata passed down to them by higher URLs' parsers) and then try to explain that to a user. **if we find what we have now is still insufficient or just unhelpfully complicated, I think I'd like to start a conversation about converting the whole thing to domain-based instead!** what a mess!
  • network report mode now reports how this stuff is navigated, for better debug in future

  • boring cleanup

  • continued my old multi-column list 'view' conversion, finishing it for all the remaining 34 lists: the 'move media files' locations list; the legacy file-lookup-script parsing system's node-editing panels; the edit account types list; the janitor process petitions list;the repair file storage locations list; the EXIF review list; all the lists in the login UI; the edit parsers list; the edit bandwidth rules list; the manage server services list; the manage client services list; the client api access keys list; the ipfs shares list; the lists in the options dialog; review vacuum data; review deferred table delete data; the debug job scheduler review panel; the manual local file import list; the file maintenance pending jobs list; the cookies review list; the network session review list; the review network jobs list; the review bandwidth usage list; the review http headers list
  • with this list refactor finally done, I can think about extending lists again. I'd like to next go for hide/showing and re-ordering columns
  • when I was poking around the AVIF stuff, I discovered regular `Pillow` seems to be getting native AVIF support very soon with the upcoming `11.2.0`. AVIF is similarly recently deprecated in `pillow_heif`. hydrus now has code to navigate this situation sanely, and as soon as the new `Pillow` comes out, hydrus will detect this and just that instead for AVIFs

• version 616

  • more media viewer stuff from a user

  • thanks to a user, we have some more media viewer updates--
  • first, if you use multiple media viewers from different tabs (especially complicated nested tabs), you can now set a 'close-media-viewer' shortcut action that does 'close-media-viewer-and-return-to-the-tab-that-opened-it'. a new checkbox under `options->media viewer` allows you to mandate this all the time
  • second, the media viewer 'eye' menu button has two new settings for setting 'always on top' and removing the window frame. these options are a little experimental and don't save just yet. have a play with them and let us know how it goes, especially on more unusual OSes

  • misc

  • you can now mix and match as many `system:hash` predicates in a query as you like. the UI will no longer consider them mutually exclusive when editing, and in the Client API, where you can force two at once, it no longer pseudorandomly chooses one to actually use
  • fixed export folders that have multiple sidecar routers that have the same file destination. previously only one of the sidecars was adding data and the others were doing nothing. (my recent 'genius' 'optimisation' code was going 'oh, that sidecar already exists, do not update it')
  • the export folder UI has a label on the sidecar section saying 'hey, export folders will not update pre-existing sidecar files, if you change them make sure to delete the existing sidecars and they will regenerate on next run'
  • when subscriptions hit their periodic file limit, they now compare the url classes of the new fetch with the oldest url class in the file log; if they differ, it no longer shows the 'subscription found x new URLs without running into any it had seen before' popup that offers the gap downloader! since in this case, the site/downloader has changed URL format, and the recommendation is to ignore the messages, I now no longer show the messages. we are doing this this week, so let me know how it goes
  • the 'search enabled' system, which on very old subscription presentation pages would allow for a search page that just presented files and had no search panel in the sidebar, is completely removed. the echoes of this thing have caused several typo problems over the years, was anti-KISS, and I intend to completely replace it with a nicer dynamic search freeze/sync system that will sync the page's current files to a fresh system:hash pred
  • the duplicate metadata merge options panel now has separate buttons on the tag service list for 'edit action' and 'edit filter'
  • the duplicate metadata merge options panel has a new label to clarify how ratings merge
  • the core Exception-handling routine can now handle an Exception with an empty string in its value
  • Errors that end up in a popup use the nicer log-printing routine to form their traceback, and they print to the log using this system too
  • errors with no trace no longer spam the stack twice

  • e621 and more

  • _tl;dr: today e621 is fixed to get tags again and I add e926 and e6ai. your subs will do some extra CPU work to catch up but it isn't a big deal_
  • the e621 sites have been changing their html recently, and it broke our parser--suddenly it wouldn't get tags any more. the good news is that someone let me know their API is excellent, so I am rolling out a fixed e621 downloader, and downloaders for e926 (their sfw alternative) and e6ai (their ai containment booru), and pool URL support too. since the API is so good, the downloader now runs extremely efficiently--all the normal 'post' data is provided at the gallery step, so e621 downloaders will hereafter only need to hit the gallery page and then they'll be downloading raw files directly. the only drawback here is the gallery parsing step eats a whack of CPU all at once, which you'll notice if you go bananas (issue #1698)
  • your e621 subscriptions will see the new direct file URLs and not be able to knit that together with the old Post URLs they already have. there is no nice technical way to get around this while keeping the nice new efficient downloader. we would normally be blitzed by the 'subscription found x new URLs without running into any it had seen before' popups that offer to open up the gap downloader, but I hate offloading that problem to user and have written in some new logic this week to detect the 'oh it is basically ok, the url format just changed' situation and silence the report in this case. luckily, your temporarily bloated e621 subs will process the 100-odd 'new' URLs they see very fast because there will be an md5 and raw file URL and e621 Post URL and tags to help it realise 'already in db'/'previously deleted'/'tag-blacklisted' without having to make any more network requests. serendipitously, this will actually help to fill in the gap of missing tags users have had over the past week or two. all users will get a popup about this on update. **if you changed your e621 subs to have a very high "normal checks" file limit, edit them before you update! 100 is proper**
  • several users were quicker than me, and I appreciate the suggestions and submissions. I made heavy use of an API parser one user made, fixing a couple tiny things and making it work for more domains, and that's what's rolling out today. you do not have to do anything--it should all just work. if you have added or played around with your own e621 parsers and today's update seems not to have worked, check out what's going on under `network->downloader components->url class links` and the other dialogs under there--worst case, if things aren't lining up, just delete everything 'e621' related and add back from the defaults

  • duplicates auto-resolution

  • thank you to the users who tried out the new system. although it works technically well, the common consensus is that it is _too_ automatic and needs A) the ability to pump the brakes and let humans better review what it is about to do and B) remember the pairs it actioned so humans can review what it just did. I am launching these features today for more testing and delaying the all-user launch of the system.
  • unfortunately, the changes I am making to the database are not compatible with the jpeg/png test rule from last week. on update, if you have the rule, you will get a popup saying 'hey, sorry, have to delete the rule now'. you can recreate it just as you did before, but the count it had of how many pairs it had processed has to be reset back to 0
  • you now choose if rules are going to be 'paused', where they do nothing, 'semi-automatic', where they will search and test but not action until the human approves, or 'fully automatic', where they will search, test, and then action entirely on their own (as before). new auto-resolution rules now start semi-automatic by default
  • beside the 'edit rules' button, there's now a 'review actions' button. double-clicking a row in the list now launches this. it opens up a panel for the selected rule with new thumbnail lists that show any semi-automatic pending pairs and the actioned pairs. they both allow multiple selection. the 'pending actions' page has approve/deny buttons, and the 'actions taken' has a powerful undo button that's wrapped in a scary yes/no confirmation. these panels are non-modal, meaning you can still use the program while they are open. the actioned audit log will often have fully deleted files so will present a blurhash or the hydrus default thumb, and any deleted files obviously won't launch in the media viewer
  • the thumbnail pair lists that show a pending action, which used to just show 'it will be this AB' vs 'could be either way around', now also says the duplicate action to be applied and an experimental summary of all content updates about to occur to both A and B. you'll see the tags, notes, urls, ratings, and archive actions--let me know how this works IRL, since I think we'll need some better sizing tech to make it all fit nice
  • a 'duplicate metadata merge options' that is set to 'always archive both' will now perform 'if one is archived, archive the other' if ran through the auto-resolution system. the auto-resolution edit UI says this explicitly. KISS answer to this sticky problem.
  • in the preview panel of the edit UI, the 'pairs that will be actioned' and 'pairs that will be skipped' lists are now collapsible
  • the cog icon on the duplicates auto-resolution sidebar now has 'reset test' and 'reset declined' to re-test all current pending/fail test results and undo all user-declined pairs
  • the main auto-resolution rule dialog now has some WOAH LAD red text at the top
  • updated the help with new screenshots and stuff for the new semi-automatic and fully automatic modes

  • boring duplicates auto-resolution stuff

  • all resolution rules and associated data will be deleted on update. sorry!
  • auto-resolution rules ditch the paused bool and now have a tri-state operation mode--paused, semi-automatic, and fully automatic. the semi-automatic does all the search and test work but queues 'ready to action' pairs for human approval. when a rule switches to fully automatic, all 'ready to action' pending pairs are sent back to be re-tested in the normal queue for KISS
  • added a new status for 'passes search, passes test, ready for actioning', with a custom pair storage table, to handle the semi-automatic mode where it prepares pairs and the human approves them
  • same deal but for 'user declined to action', for when the user does not approve
  • figured out the pipeline for this
  • added a table to log AB file pairs actioned. it records the duplicate action and the timestamp also. the 'pairs resolved' count is now generated from this live and stored in the normal count cache table
  • figured out the pipeline for this record
  • I vacillated and wrote some code to make the history try to 'live sync' to current data, but in the end I decided to go full non-changing audit log. if you dissolve a duplicate group and re-action the files with the same rule, the log will have two entries
  • if a rule's selector changes after editing, all pairs in the 'failed test' or 'ready to action' queues are now reset to 'passes search, ready to test'. relatedly, pair selectors can now equality-compare themselves to each other
  • the thumbnail pair lists that list duplicate merge summary data now calculate that third column in an async thread. previously this happened in one big block before the list was populated, but in future selector work will be high CPU (think comparing image similarity live), so it now happens on each list row as it comes into view. there's still a little more work to do here in the preview panel
  • refactored the new thumbnail list classes out to their own file and renamed some things to be more generic (for use in the new 'review actions' panel too)
  • if a user tries to edit the auto-resolution rules while any of the new preview windows is open, the preview windows are closed
  • the thumbnail lists now check if media it wants to open in the media viewer is local lol
  • cleaned up the 'reset search' and 'reset test' maintenance calls and improved efficiency
  • refactored how I move pairs from one database queue to another and cleaned it up a bit. a lot of maintenance and other reset stuff now all goes through this one location
  • a bunch of misc refactoring and cleanup of the auto-resolution db module

  • unit tests

  • added unit tests for the semi-automatic auto-resolution rule queueing, queue-fetching, and approval and denial processes
  • added unit tests for the audit log after all other unit tests commit actions
  • added a unit test to ensure changing the search resets searched pairs back to 'not searched'
  • added a unit test to ensure changing the selector resets tested pairs back to 'match search but not tested'
  • added a unit test to ensure changing from semi- to fully automatic resets 'ready to action' pairs back to 'not tested'

  • misc boring stuff

  • added a convenience function to the media results database module to fetch pairs of media results
  • made my static box change its size policy to fixed when it is collapsed--it now tucks itself away where before it was expanding to any available space
  • fixed the new auto-resolution unit tests for python 3.12+. thanks to a user for reporting--I was using the deprecated `assertNotEquals` instead of `assertNotEqual`
  • fixed some Mr Bones crazy text alignment and he now tells you in the duplicates panel to set to "all files every imported or deleted" to get better "all-time" numbers
  • Mr Bones now gives more accurate 'total files in alternate groups' numbers and also says the total number of alternate groups. previously, he was basically adding a bunch of duplicate files in there that were standing on their own and hadn't actually been set alternate to anything
  • updated my 'install help' Linux tab Wayland section to say 'yeah, best solution is X11 for now' with the two ways we know of getting that to work within Wayland. that's my Wayland policy going forward

  • client api

  • the Mr Bones call on the Client API now has a 'total_alternate_groups' key, reflecting the new number in the window
  • Client API version is now 79

• version 615

  • duplicates auto-resolution brief

  • the system is ready for advanced users to try! there is one simple static rule available. check out the help https://hydrusnetwork.github.io/hydrus/advanced_duplicates_auto_resolution.html , go into the UI, and try out the suggested pixel-perfect jpg & png rule. I want to know if--
  • - it all makes sense to you
  • - where it runs smooth
  • - where it runs garbage
  • - any errors?
  • - out of interest, what do you get? Of ~800k potential pairs, I had ~6,000 jpg/png pixel pairs, resulting in ~4,700 total actual processed pairs (numbers shrink because multiple good files can share the same bad file). speed was bleh in the preview viewer (about 30 seconds to load the preview numbers) but nice when doing work: only a second or two to save the rule and then ~20k files/s in the search stage and 10 files/s in the processing stage. about 7 mins to ditch 7.5GB of Clipboard.png, hooray

  • duplicates auto-resolution

  • fleshed out the help page here: https://hydrusnetwork.github.io/hydrus/advanced_duplicates_auto_resolution.html it is linked in the main help directory, too, under 'advanced'
  • made it so you can double-click or enter/return any pass/fail test row in the auto-resolution preview panel to open that pair in a normal media viewer
  • added `work on these rules during idle/normal time` to the cog button on the auto-resolution sidebar tab
  • wrote up 'work hard' functionality and wired up the button--however I think I might remove this, since the system works well enough on its own. let me know what you think
  • reworked the preview panel to have a two-stage search. it spends a whack of CPU time fetching the total count of the search, and then the sample part works faster afterwards and can be hit over and over. it still sucks and I have another idea to speed things up, we'll see
  • updated the main duplicates auto-resolution maintenance routine to do the search step in blocks of 8192 rather than trying to do everything at once as soon as a rule is added. I hesitated doing this earlier, since the large one-time init search is more efficient, but we are going to be doing incremental searches in normal operation, so making small search chunks work well is not optional. I think the biggest potential weakness of this whole system is going to be incremental searches on new potential pairs when the potential duplicate pairs search of the auto-resolution rule has significant per-search overhead, but which searches will those be? sounds like we'll find out in the coming weeks. thankfully, jpg/png pixel dupes seems to scale excellently, so we are good for today

  • misc auto-resolution stuff

  • reduced flicker on the main 'auto-resolution' review panel as it works a real rule
  • fixed a 'hey I guess the auto-resolution preview panel is 1200px wide now' bug on certain rule-fetch validation errors
  • renamed 'A will pass' comparison UI labels to 'A will match'
  • the preview panel lists now scroll one row per scroll (it was doing 3 before, lol, and the default height is ~2 rows)
  • the auto-resolution preview panel (particularly the 'will be actioned' list, with its third column) should size itself better if you have large thumbnails
  • if the auto-resolution preview panel had results but the search changes and now we have no resullts, the test result lists will now appropriaterly clear themselves
  • the preview result lists now generally clear nicely before a new search is started, so if something actually goes wrong, you don't have old test results hanging around
  • fixed incremental potential duplicates search on non-pixel-duplicate searches

  • unit tests

  • wrote unit tests for the `MetadataConditional`
  • wrote unit tests for all the predicate types that can work in the `MetadataConditional`
  • wrote unit tests for the `Selector`
  • wrote unit tests for the one-file `Comparator`
  • wrote unit tests for auto-resolution rules: for editing at the db level; for syncing to existing, new, dissolved, and resolved potential duplicate pairs; performing search; performing resolution

  • misc

  • fixed a typo bug that broke the maintenance job that resets all potential duplicate pair search. it was related to the recent auto-resolution integration
  • fixed a typo bug that was stopping the 'review accounts' repository admin panel from opening
  • added an FAQ about the extensionless files that appear in your file storage if you sync with a repository
  • added `LINUX DEBUG: Do not allow combined setGeometry on mpv window` to `options->media playback`. if you have crashes on X11 in v614 when zooming mpv windows, give it a go and let me know what happens
  • thanks to a user, added a note to the install help that if you are on Linux & Wayland, adding the `WAYLAND_DISPLAY` environment variable, which forces the program to run in XWayland, seems to relieve many UI bugs (issue #1695)

  • boring cleanup

  • cleared out a surplus entry in db init related to duplicates auto-resolution
  • tweaked the 'don't show hovers/hide cursor if a dialog is open' tests in the media viewer, ignoring them if we are the child of one lol

• version 614

  • misc

  • the new `system:tag (advanced)` edit panel now has a 'write/edit' tag autocomplete to help you quickly enter what you want to search
  • the `system:tag (advanced)` predicate now works in the system predicate parser. everything it produces should be pastable back into the tag autocomplete and it'll all just work (I believe even crazy situations, but let me know how you get on). it is somewhat error tolerant, so you can type just 'pending' instead of 'status is pending' etc.., but it is best if you get the colons, commas and quotes correct
  • reworded the labels in `system:tag (advanced)` to say 'ignoring siblings/parents' rather than just 'siblings'. I was thinking about this the wrong way when I first implemented it and forgot to realise and mention that when searching in the 'storage' domain, you are missing tag mappings that would be implicated by parent relationships too
  • thanks to a user, we have two new QSS styles--'catmocchin blue' and 'catmocchin lavender'. there are additional style and colour suggestions inside the QSS files themselves
  • the default image cache size under `options->speed and memory` is raised from 384MB to 1GB, which means by default the cache will hold on to a ~12,600x7,000px image and prefetch a ~9,700x5,500px one. it was previously tuned for 4k, but we are going to push a little further
  • `speed and memory` also lets you raise the prefetch cache percentage up to 25% (from 20%) (issue #1693)
  • I discovered that the bug in the media viewer where a static image will do a flickery resize--where on zoom it will move to position and then scale in two discrete frames--is triggered by loading the mpv window in that media viewer! it seems like mpv sets some deep 'immediate render' flag on the window. I hacked around a bit and believe I have fixed this bug. a rewrite of the entire media viewer layout system remains pending, but I think we fixed this very annoying thing!
  • I may have also fixed some other change-media flicker too, let me know how you get on
  • the 'macOS window position fix test' debug mode is now default behaviour. if you noticed your dialog windows were moving down like 26 pixels on every open in macOS, it should be better now! (issues #1681, #1673)
  • fixed a bug when searching files with 0 width or height (this is mostly a legacy issue) with a file-sort set to image ratio
  • if you try to run `profile mode` on python 3.12+, you now get a popup saying it is broke atm and hydev will fix

  • new media viewer tech

  • thanks to a user, we have some great new media viewer zoom and display options
  • there's a new zoom icon button in the top hover that lets you set some new zoom types: default for filetype, 100% zoom, canvas fit, fit horizontally, fit vertically, and canvas fill
  • also, we now have 'lock current zoom type'! so, you can switch to 'view this image as 100%', and that will stick for the next file as you browse back and forth
  • also, there's 'lock current pan'! useful for comparing duplicates at high zoom
  • _and_ there's 'try to lock current size', which copies my duplicate filter's 'lock zoom amount' tech for the normal media viewer
  • these three 'lock' options are saved, not per-media-viewer-session
  • you can say in `options->media playback` what zoom type you would like the viewer to default to. the default is 'default for filetype', lol, which I generally recommend, but you can set to always override all filetypes with 'canvas fill' or something if you like, particularly, say, on the preview viewer
  • also, under the 'eye' icon, or `options->media viewer`, you can now set the tags hover and/or top-right hover to not appear on mouseover!
  • the 'media viewer' shortcut set gets actions for the new zoom types, a new three-way zoom switch, and a new 'recenter media' action
  • I fixed a couple things with this for weird stuff like excepting audio and open externally panels. I also span out the duplicate filter zoom maintenance into its own thing and added it to the new menu as the 'lock current size' choice, reworked the options stuff a little to fit in with the existing per-filetype zoom settings, added some checkboxes to the menu for feedback, added the 'recenter-pan' action, and since graphics design is my passion I made a new icon for the advanced zoom settings icon button, yes it is a cog icon placed expertly over a magnifying glass
  • I think I fixed some zoom bugs in the duplicate filter with 'open externally' and audio panels

  • duplicates auto-resolution

  • everything went great. should be launching the initial test for advanced users next week
  • extended the new preview panel a bit--added a 'only sample this many' number widget, defaulting to 256, so you only start looking at a fast preview of the potentially tens of thousands of results; made sure the results always sort the same way (pseudorandom, but fixed); and added a third column to the 'pairs that will be actioned' list to show if the AB pair is fixed as you see or could be either way around
  • wired the potential duplicates storage module up to the auto-resolution storage module. when a potential pair is added or removed, the auto-resolution rules are now synced in the same transaction
  • buffed the 'add pairs' tech here to ensure even if the two modules get desynced somehow, the auto-resolution guy won't add dupes by accident
  • hooked the 'delete orphan rules' and 'fix orphan potential pairs' auto-resolution maintenance jobs up to the cog icon menu button in the auto-resolution sidebar panel
  • wrote some async code to handle waiting for any current work to finish before launching the edit dialog, and only allowing one edit dialog at a time
  • added a refresh button to the review panel
  • fixed up and finished the main 'set rules' pipeline, including generation and propagation of rule_id and counts cache
  • reworded the rule status summary text to put pertinent info at the front
  • hooked up the main search and resolution worker db calls
  • fixed a ton of stupid typos in the auto-resolution db code
  • enabled the UI, tried it out; the whole system works!
  • the UI is enabled for today, but the edit rules dialog will not save anything. non-advanced-mode users can also see the new tab now. I will take another week to write up some unit tests and help and do a more complicated IRL test. I also didn't have time to add the media viewer so I'll make sure that gets done. otherwise, we appear to be basically ready here. I'm feeling good about it, but I still want to be quite careful

  • boring cleanup

  • I cleared out some bad old and duplicated canvas zoom code, and I migrated some variable handling to use the smarter stuff in the new commit
  • replaced some laborious resolution validity-testing code with a single call in media result
  • cleaned up how potential duplicate pairs are deleted in the main duplicate files db module. it now happens through a handful of central locations rather than spammed all over the place, so the signalling to the auto-resolution module is a good bit simpler
  • fixed the permalink id in the client api help for `/get_files/thumbnail_path`. also fixed the example requests, which were also a sloppy copy/paste job from the 'thumbnail' command haha
  • fixed some `None` to `null` in the client api help response examples

• version 613

  • misc

  • I think I have fixed the crazy flickering dialog resize bug where if you resize certain dialogs shorter or thinner than their parents, they might flicker throughout your resize motion between the size you are moving to and the height/width of the parent. there's still some occasional flicker due to some other legacy fuzzy padding hacks, but things are better
  • fixed some bad grammar for the namespace hide/show options in `options->tag presentation`, and split the non-namespace rendering settings out to their own box
  • when files are imported, any existing 'media result' object for the file is now properly updated with the new file modified time. you usually wouldn't notice this, because the media result typically isn't loaded until after import when we need to show it on screen, but in advanced cases where you were re-importing previously deleted files you are looking at or doing Client API work where you did a URL lookup before deciding to import, and the media result object from the URL lookup was still in memory, the 'media result' was not getting the timestamp and subsequent inspections or edits of the modified timestamp were failing as a result until the media result had a chance to decay out of its cache and be reloaded from disk
  • made some ratings update and sort code more safe against ratings service deletions
  • url classes now allow 'keep extra parameters for server' if you have an API/Redirect URL Converter. I _think_ the reasons for this originally being prohibited were only a fear, not anything grounded, but let's see how it goes

  • system predicate for advanced tag search

  • _after talking about it with people recently and having some good ideas, I decided to just push and try to knock this out. it is pretty clever and may have bugs, so let me know how it goes_
  • added a new `system:tag (advanced)` predicate, which does sophisticated tag searches. you enter a tag as raw text and can then--
  • specify which tag service it should search on, if different to the current tag context
  • specify if it should include siblings or just be the raw tag on the 'storage' domain
  • specify whether you should search current, deleted, pending, or petitioned mappings
  • specify if the file _should_ or _should not_ have the tag mapping
  • I ran out of time to implement system predicate parsing, but I planned the pred's label to support this and will try to get it done next week
  • searches that include unified or deleted file domains or any other crazy stuff may take a very long time to return, but the underlying tag search code in hydrus is pretty robust, and extending it to the newer domains all just came together, so I think everything works. it should respond quickly to the cancel search button if it takes too long

  • duplicates auto-resolution

  • I had more success than expected with the new preview panel and I think we are now about two weeks away from the initial user test of this system
  • the edit duplicates auto-resolution rule dialog now has a live preview panel. it loads the pairs of a rule's search and then does the comparator/selector test to determine which pairs pass or fail, showing you the counts at each stage
  • the panel also shows the pass/fail thumbnail pairs with some new 'thumbnail table' widgets I wrote. these load thumbnails asynchronously and should scale up to thousands of pairs no prob. I am really happy with how it worked out, as this was the thing I was dreading would take weeks of rewrites, but I figured something out. it is ugly, as always, but it works
  • every time you show the panel page, it checks if the rule has changed. if search has changed, it re-runs the whole search, and if only the selector has changed, it just does a re-test
  • next week I'll add the ability to launch a particular pair into a standard Media Viewer so you can 'look closer' on the preview, and then I'll be doing final integration and testing. we are on the home stretch now!
  • _if you are an advanced user and you've poked around the duplicates auto-resolution UI previously, you might like to check out the preview panel this week on the default jpg vs png rule--does it all seem to load up sensible results, even though you can't zoom in yet? any errors? I'm guessing that loading thousands of pairs is going to be super slow, so what would be a good number to sample--256?_

  • boring cleanup

  • reworked the client thumbnail cache to work on the simpler and lower-lever `MediaResult` rather than the UI-level `Media`
  • fixed a couple areas where file maintenance jobs were being scheduled using the `Media` object
  • fixed some old areas within file maintenance where media objects had the wrong name
  • reworked and cleaned the main GetThumbnailPath RegenThumbnail file routines to similarly only work and talk about `MediaResult`
  • fixed a dumb typo in the as-yet-enabled `MetadataConditional` edit-panel
  • updated the faq with answers for why I work alone, do weekly releases, and use weird versioning
  • fixed a couple places where I had accidental `/n` in some label text lol, thanks to the user who pointed this out

• version 612

  • misc

  • I added 'you can scan for this' flags for JXL for rotation EXIF, has EXIF, has non-EXIF, has transparency, and has ICC Profile, which we forgot to add when first rolling out JXL support. all existing JXL files will be scheduled for rescan of these properties on update, so if you have any wrongly rotated JXL, they should fix themselves soon after update (issue #1684)
  • the 'duration' duplicates filter comparison statement now says the +/-% difference in duration. if it is only +1%, it is now clearer
  • if a server gives a 429 (Too Many Requests) with a 'Retry-After' response header, hydrus now obeys this rather than its default server retry time as set in `options->connection` (issue #734)
  • added `help->debug->debug modes->macOS window position fix test` to test a fix for macOS windows always getting delta y position on init

  • drag-cancelling tech

  • the new drag-cancelling tech was false-positive cancelling fast drags where you moved the mouse outside of the window 200ms after the initial click press. this is fixed
  • the new drag-cancelling tech no longer occurs on macOS, since it was raising an error. it seems due to system policy you can't programatically cancel a drag on macOS in Qt (issue #1685)

  • sidecars in export folders

  • an export folder with sidecars will now fill in any sidecar gaps on every run. if a file should have a sidecar but currently does not, the sidecar will be generated (issue #1682)
  • an export folder set to 'synchronise' (which deletes files from the export destination if they are no longer in the export folder's search) will now be more careful not to delete sidecars

  • some running from source stuff

  • for users who run from source, the 'test' `PySide6` is updated from `6.8.1.1` to `6.8.2.1`, and the `qtpy` from `2.4.2` to `2.4.3`
  • added a `open_venv.ps1` script to easily activate the venv in powershell

  • duplicates auto-resolution

  • _my work this upcoming system grinds on. I successfully did the thing I stalled on a couple weeks ago, and now the only big thing I still have to do before advanced users can try it is a preview panel and a billion unit tests_
  • the database modules are live! they boot and will initialise their (as-yet empty) tables on update to v612
  • the daemon is now live! it runs its full mainloop and consults the database modules for work
  • rewrote some daemon-db interaction, mostly for KISS and deduplication of responsibility
  • figured out some tech to reset duplicates rule search progress on edits and user command
  • simplified the daemon workflow so that rules hold on to cached search counts and report their own search status
  • fixed some stuff in rule-setting and status fetching
  • the auto-resolution rule object is now serialisable and _could_ be saved to disk if I enabled the edit UI
  • rules now report their correct search status, no longer a placeholder
  • the daemon now reports the correct running status, no longer a placeholder

• version 611

  • misc

  • I am pretty sure I fixed the long-time 'hey why did the scrollbar on this page of thumbs reset to the start position sometime while the page was hidden?' issue. an importer that adds new rows of thumbs in the background should now retain its scroll position. a window resize or a mass file removal event will cause the scrollbar to stay at the same relative num_rows vertical position, which we may ultimately want to be more sophisticated, but this is a good step for now. let me know if any specific situation still resets the vertical position
  • fixed an interesting bug with two of the Drag and Drops in the program, when you DnD a page tab or a selection of thumbnaills, where if you released your left-click at just the right moment (about 100ms after click press), you could get a DnD state that persisted despite no mouse button being down. a subsequent click would clear it. it seems that the DnD was spawning while the click release was already behind it in the event queue. the program now recognises this state and cancels the DnD
  • I believe I fixed an issue with the `apply image ICC Profile colour adjustments` entry on the media viewer's 'eye' menu button, precisely when you were looking at a file. the static image's private tile cache was not clearing itself correctly, and not calling for an update after the clear, so old yes/no ICC-profile-adjusted tiles were hanging around and sometimes poisoning future calls. the whole system clears itself properly now and seems to flip good every time
  • I believe I have fixed a macOS issue (and I think X11, too, if you have a taskbar on the top of the screen) where the hydrus window position-recording system would move your windows down a little way on every save/load cycle
  • all the 'write/edit' tag autocomplete inputs that have a list above them now send a copy shortcut to that top list when there is nothing selected in the text box and there are no results in the list below. so, if you open the 'manage tags' dialog, and the first thing you do is hit ctrl+c, you'll copy everything in the big list of existing tags. hit ctrl+shift+c, and you'll get the parents too
  • all the hardcoded 'copy' shortcut detection now happens through the same location and counts ctrl+c or ctrl/insert
  • if the 'is user ok to ok/cancel' dialog pre-close checks raise an error, I now have a catch for it. it says 'hey user, these checks threw an exception, please contact hydev, the data was probably broken so here is what to do next depending on whether you were oking or cancelling'. previously, you could be stuck with an unclosable dialog and have to quit the whole process
  • same deal but for the dialog 'is the data in the panel valid?' check--there is now a special catch and it tells you what happened and what to do
  • fixed a regression from last week's "make sure the new note is focused when the current tab changes" call, which was throwing a spurious error when you deleted the last note (issue #1679)
  • fixed an issue with my mpv rewind-spam catching code when you force it to load a file without any duration

  • file sort

  • if you change the file sort on a search with an explicit system:limit, and the sort is one that can operate at the database level, the search will now refresh
  • added a checkbox to `options->file search` to turn this off if you prefer
  • the file sort widget now has a tooltip saying whether the current sort type is simple enough to work with system:limit

  • stopping accidental page close

  • if you try to close a downloader that has any items in it, you'll now be asked if you are sure. this doesn't apply to session exit, just manual user close events
  • you can turn this confirmation off under `options->gui pages`
  • another checkbox under `gui pages`, this time default off, allows you to confirm all page closes. this mostly catches middle-clicking single page tabs, be that file search pages or 'page of pages'
  • the routines that delete multiple pages in one step 'close other pages' etc..., now do the 'is user ok to close these' in one combined step for all closees before closing anything. previously it tested and closed each in turn, and a cancel event broke out of the loop
  • various 'close these pages?' messages now correctly add up the number of sub-pages in 'page of pages' to be closed

  • long-time bug with non-updating media viewer is fixed

  • for ages we've had a weird issue where the media viewer, on some random file, will suddenly update when you click on ratings and things. the content signals are going through, and you can watch the file's respective thumbnail in the main gui getting archived or whatever, but the media viewer will not show it. this is finally fixed! it was basically a rare maintenance update issue, where if, for instance, a 'regen file metadata' job occured on a file in a media viewer's list after the media viewer was spawned, the media object in the viewer list was falling out of sync; able to send content updates but no longer hooked into the update pipeline so it never received them back again
  • this is fixed for the 'force filetype' action. this job now propagates to the UI much faster than before, no longer needing a full media result load
  • this is fixed for the file maintenance jobs that regen file metadata, check for 'has icc profile' etc.., and regen pixel/blurhash. all these jobs now propagate in the same way much faster to the UI
  • this is fixed for the file maintenance job that refetches the file's modified time from disk. same deal it now updates in UI faster
  • also fixed in the case where a previously known-of file is imported and we discover our old file info was incorrect, and also this file was in view so we need to sync the media result. sounds like a crazy situation but that's exactly when you need this to work

  • tag context in media

  • the thumbnail grid is now aware of the current tag context of the search (where you set 'all known tags' or 'my tags' or whatever in the autocomplete dropdown). `system:number of tags` now follows the current tag context

  • shortcuts

  • in an experiment, you can now set the modifier keys as their own shortcuts, with no accompanying 'real' key. it'll mark it as 'ctrl+nothing', 'shift+alt+nothing', etc.. seems to work, but I can't promise it won't go crazy in the wrong place
  • the keyboard shortcut widget, where you enter a shortcut, now uses a shortcut catching routine that is one level higher than before, which matches the actual shortcut processing system that's in use. there's a chance that if your Qt hardcode maps some shortcut, not allowing it in a text edit box, it'll work now (issue #1674)
  • the keyboard shortcut widget no longer shows its caret. it also takes keyboard focus more reliably when the edit shortcut dialog boots. I kind of like this but also don't, let me know if it is jarring
  • the 'shortcut report mode' will now say some information about unmappable key presses
  • I have prepped some render code to show shortcuts on macOS in the form ⌘Z, but I have not enabled it yet because I don't know enough about macOS. if you want your shortcut labels to look like this everywhere, let me know. but if shortcuts should only look like that in menus, and it isn't appropriate for the shortcuts dialog and debug labels and stuff, I'll put it off until we actually have menu shortcut labels lol

  • parsing

  • I added a 'STATIC' formula type. it just spits out the same static text, every time, and repeated n times if you need it
  • added a little help documentation for the new STATIC formula type; the edit panel links to it like for the other formulae
  • if your parsing test panel gets something that doesn't seem to be html or json, it no longer puts the goofy quote marks around it

  • boring code cleanup

  • did some misc refactoring in the page close testing code because the method names were whack
  • cleaned some sidebar and autocomplete interactions to work with 'tag context' instead of 'tag service key', which means these guys are now sending a richer object with more tag display info in it
  • cleaned up some duplicate variable handling all through the sidebar stuff
  • a bunch of redundant location context emit spam from the sidebar is deleted. I don't think this will break any weird thumbnail grid or previuw canvas location stuff , but we'll see in the edge cases
  • cleaned up how query autocomplete dropdowns and sidebar panels were communicating search state changes
  • ratings that fail to draw for whatever reason now do so more safely
  • removed code that did the old and canvas-desyncing media object update routine

• version 610

  • misc

  • files with an average colour with less than 3% saturation in the HSL colour space (i.e. completely greyscale, or otherwise averaging to grey) are now propagated to the end of the Hue sort, whether red or purple first
  • hitting ctrl+c on a taglist with no selected items now copies all items
  • updated the system predicate parser so it can read 'number of pixels'. previously, it could only handle 'num pixels'. this means you can copy/paste a 'system:number of pixels' pred back into the autocomplete and it should all work
  • fixed a problem with `system:num file relationships - (test) "not related/false positive"`, which was only every returning the results of one not-related alternates group, rather than all that matched the test
  • made some 'click below to copy to clipboard' menu labels bold
  • the client now says 'tag mappings' explicitly, rather than 'mappings', in a bunch of UI labels like shortcut action descriptions

  • notes

  • the 'start editing notes with the text cursor at the end' setting under `options->notes` now applies to all notes in an 'edit notes' panel with multiple notes
  • clicking a new note tab in 'edit notes' now immediately focuses the underlying note so you can start typing
  • if you have made changes, the 'edit notes' panel now confirms if you want to cancel. let me know if this produces false positives--there's some 'note cleaning' stuff I don't account for

  • file viewing statistics

  • the new 'client api' views are now integrated into the UI for search, sort, and view
  • the way file viewing stats are presented in the media right-click menu is simplified. the stuff under `options->file viewing statistics` is down from five different options to two (show the views summed with a stack of the components in a submenu, or just show the stack of separate count types up front), and there's a new checkbox list with media, preview, and client api views so you can select what you want to see
  • sorting files by media views now obeys the above 'what you want to see' setting, summing the respective values. previously it just did media views
  • `system:file viewing statistics` now allows for selection of 'client api views'
  • `system:file viewing statistics` now renders itself in the form `system:views/viewtime in (domains) (operator) (value)`, where 'domains' is a comma-separated combination of `media`, `preview` and `client api`. previously they were `system:media/preview/all views/viewtime (operator) (value)`
  • if domains is left blank, I substitute the above 'what you want to see' option, so you can now just type `system:views > 5` and you'll get something reasonable
  • the old forms still parse but are unchanged and won't support client api
  • added some new unit tests for this
  • as a side thing, I did the same pixel-perfect-height update that I did for multi-column lists to most of my checkbox lists across the program, including for canvas view selection around here. they just fit better now
  • I overhauled some file viewing stats db search code. the predicate still holds some stupid variables that I'll want to rework one day, but the meat of the db search code is now ok

  • advanced parsing logic and overhaul

  • subsidiary page parsers have a new 'sort posts by source time' checkbox, default off, which sorts what this subsidiary parser chops up according to any parsed source time, newest first. use this if you are, obviously, parsing things that each have a source time you'd like to sort by but which come in some other order, e.g. you have two individually sorted pages of results in one document and you'd like to interleave them
  • wrote up proper objects to hold 'parsable content description', 'parsed content', and 'parsed post'. previously this was a big mess of tuples upon tuples and 'all parse results' vs 'whole page parse results' semantic sludge
  • refactored `ClientParsing` to a new `parsing` module, added `ClientParsingResults` to handle the new objects, and split off old file lookup stuff to `ClientParsingLegacy`
  • all parsers now use the new objects to report what content they can parse
  • all parsers now use the new objects all along the parsing pipeline! the whole thing is done and much nicer
  • deleted a ton of old bad code that was navigating this before
  • deleted some old bytes/str error-handling back from the python 2 days
  • when a gallery url is worked, it now reports fully, in the gallery log note column, how many sub-gallery urls were parsed, and how many were previously hit this run, and how many next gallery page urls were parsed, and how many were previously hit this run, and whether a url class extrapolated next gallery page url was generated, and, the same deal, if it was previously hit this run. previously, this guy mostly just reported the latest interesting thing to happen--now it is comprehensive
  • fixed an odd bug where the gallery worker could add a sub-gallery urls that pointed to the API redirected URL that created it
  • fixed an odd bug where the gallery worker could skip the 'seen this url before in this parsing run?' check for an auto-generated 'next page' gallery url fallback step
  • cleaned up the gallery url 'work' code significantly
  • fixed a note parsing issue in the downloader where note texts were not always being cleaned on parse
  • overhauled the subsidiary parser test panel to use the actual code as-in the subsidiary parser, rather than a facsimile
  • got the page parser and subsidiary page parser test panels to recognise empty input rather than outputting errors on failed conversions of the empty string lol
  • all the 'raw data'/'post conversion'/'post separation' tabs in the parser test UI now show up to 64KB of data. previously, the latter two were still capped to 1KBs
  • fixed some bad layout in the edit subsidiary page parser panel
  • cleaned up some lazy variable names in the subsidiary page parser code
  • cleaned up how subsidiary page parsers initialise their edit panel
  • fixed a layout issue in the 'edit login step' panel where there were now two nested 'content parsers' static boxes

  • some subscription logic and UI

  • cleaned some of the subscription gallery search logic, especially the 'have we caught up to where we were despite the (large) gallery page not being finished yet?' test, which has a couple of false positive edge cases solved
  • cleaned some duplicate code out of here
  • subscription queries now have an editable 'compaction number', which governs how small they should trim themselves to. this was previously fixed at 250 items, but is now changeable for advanced debugging purposes under the edit subscription query panel
  • if you have a very large periodic file limit or the site produces a huge gallery page, the compaction number will now increase dynamically to ensure the subscription maintains a healthy cache of urls to consult for the size of our job
  • the edit subscription query panel gets a layout overhaul. the stack of widgets is now grouped into boxes
  • the `options->downloading` page now specifies that the two checker options buttons are for the subscription/watcher _defaults_
  • as a side thing, watcher check logs now record up to 500 entries before removing old items. used to be 100

  • weird thing

  • the tag autocomplete searches across the program that manage a `read/search` context (as opposed to a `write/edit` one), will now pass a keyboard copy event to their top list if there is no selected text in the text input and there are no results in the search list below. actually getting into this situation is tricky, but this is really prep for a future overhaul of the write context, where no results is the default situation on an empty input but which cannot actually talk to its top list yet

• version 609

  • macOS app now comes in a zip

  • the macos-13 runner that we recently moved to has been taking way too long to build the App DMG. under ideal conditions it will take ~6 minutes, but sometimes it will be 45+. this is an unfortunate and widespread issue without an excellent fix
  • thus, the macOS App is now distributed in a zip (that takes about 30 seconds to build and is the same size as the DMG lol)
  • unfortunately this means we can't have the Applications shortcut or the ReadMe; you'll be double-clicking the zip to extract the App, and then dragging it to your Applications yourself
  • I have updated the 'getting started - installing' guide with the 'allow unsigned apps' stuff from the ReadMe
  • I am increasingly of the mind that macOS users should consider running from source. the App is built for intel chipsets, so if you are on Silicon, you definitely should: https://hydrusnetwork.github.io/hydrus/running_from_source.html

  • misc

  • INFO: reducing the 'too many events' mpv error-handling hook worked well. if you saw a bunch of these in the past few weeks, these were almost certainly false-positives. the files were likely not broken, they were just taxing your computer for a bit and the error was firing accidentally. if you set up a special workflow to isolate or deal with these files, please undo it
  • the new 'group by namespace (user)' list widget under `options->sort/collect` now accepts `:` to represent 'all other namespaced tags' if you would like to put them somewhere specific
  • fixed a typo in the new duplicate files 'has duration' comparison strings. it said 'the other has duration, the other does not' when the current file had no duration
  • the score for duration now scales from 1 to 50 depending on how much longer a vid is than the other on millisecond longer = 1, twice as long = 50 (cap). this is all still just hardcoded stuff I'm playing around with, and I expect it to formalise to something completely user-customisable as the tools from duplicates auto-resolution firm up
  • added a little section to 'help my db is broke.txt' talking about the sqlite3 terminal and how to get it on non-Windows

  • colour sorting

  • after talking with some users, and learning that our blurhash provides the average colour of image thumbnails, I went bananas and added five new 'average colour' file sorts--
  • `lightness` - sorts light-to-dark in the Lab colourspace
  • `hue` - sorts the standard red-orange-yellow-green-cyan-blue-purple colourwheel/rainbow in the HSL colourspace
  • `balance - blue-yellow` - sorts blue-yellow chromaticity in the Lab colourspace
  • `balance - green-red` - sorts green-red chromaticity in the Lab colourspace
  • `chromatic magnitude` - sorts the combined chroma value, which is basically 'is it grey or colourful' in the Lab colourspace
  • this works on any file with a thumbnail!
  • they all work with system:limit, also!
  • and the Client API--new `file_sort_type` definitions are in the Client API reference
  • I am extremely pleased with how this all worked out. there are some weird results because it is average colour, rather, say, than a carefully selected 'primary colour', and I think it could do with some more work, to perhaps, for instance, push remarkably grey images to the bottom of Hue sorts, but this really works and it simply looks cool to do and then scroll through. it is a good base and proof of concept for my original plan to make colour histograms for files and distinguish between images that are muddy all over versus white and red. let me know where this succeeds and fails, and we'll see what improvements we can think of

  • janitor stuff

  • in the petition processing page, when you process a petition, either the normal way or via the mass approve/deny buttons, the 'currently highlighted petition' moves to _after_ the current one (or the next available one if you process multiple at once). previously this was resetting to the 0 position on every mass approve/deny. it should ripple down a bit more naturally now
  • when you set to apply multiple petitions at once, the pertinent list rows now instantly to show 'uploading...' status
  • I wrote a whack of test code that loads up and processes fake petitions for the petition page, so I can test these workflows much easier now
  • I fixed the 'modify account that uploaded this tag' for tags that include the '+' character. sorry for the long-time trouble--it was a stupid GET args parsing issue I missed in early code

  • boring code cleanup

  • broke up the 'ManagementPanels' monolith, refactoring it to a bunch of smaller files more under the 'Sidebar' name
  • renamed the ambiguous 'management type' to 'page type', and 'management panel' to 'sidebar', and 'management controller' to 'page manager' across the program
  • cleaned up some old static superclass calls in here to `super()`
  • brushed up the code generally around here, ditching some bad ideas and scope design
  • fixed a small instability trigger in the main file query routine
  • cleaned up some more inartful controller scope code in page and downloader UI
  • cleaned some ugly `QLineEdit` initialisations across the program
  • updated the server requirements.txt, which isn't used for anything but a reference, to use the newer OpenCV; I just forgot to update it a while ago

  • pyproject.toml

  • a user made a 'pyproject.toml' file. this is basically a cleverer requirements.txt that allows for better dependency options. I learned a bit about these files this week and I quite like them. pip can handle them too, so this would be one way to simplify my setup_venv scripts to a single call and allow users to set up simple custom automated situations
  • so, just for now, there's a pyproject.toml in the base directory. if you use `uv`, feel free to play around with it. I will manually keep it updated and version-synced to our requirements txts, and I wouldn't be surprised if I switch to it as the master soon. I'm going to do some other work and research and testing first though

  • duplicates auto-resolution

  • I did a little work on the daemon, but I didn't feel great about it tbh. I know roughly the various things it needs to do, but I'm not firm on what should be the correct orientation of everything. I'll keep at it, and I may do more UI so I have better feedback while I am doing final integration here

• version 608

  • mpv

  • my ill-fated mpv-choking error hook, which tries to catch when MPV is having a 100% CPU event and unloads the file in a big fuss, and which has unfortunately caused a whole bunch of false positives, is now flipped on its head. instead of catching all instances of the log output and trying to exclude 'oh that's fine' cases, the hook now precisely targets files that go irrevocably 100% CPU on loop, specifically by recognising files that reload too frequently in one second for their duration (e.g. a normal vid looping twenty times in one second). mpv will be back to being juddery sometimes, but whatever, that's better than what I was doing
  • the new 'DEBUG CAREFUL: Allow "too many events queued" files' checkbox no longer does anything and is removed
  • also, the mpv emergency dump-out code is back to being per mpv player. previously it would unload all players the client was managing
  • my mpv player now uses a little less CPU when mouse-drag-seeking

  • misc

  • I am changing the tag sort logic such that if you group by namespace, the namespace order stays as it is, no matter the underling asc/desc order of the actual sort. in simple english, this means that creator tags stay at the top whether you say 'most first' or 'fewest first'. we had success with the 'namespace (user)' just recently, and after talking about it with some guys, I think I agree that doing a full reverse of the whole list is not useful, and it isn't, fundamentally, what grouping is. let's try it like this, see how we like it
  • the various multi-column lists across the program, which generally set their minimum and initial height by a 'num rows' suggestion, now do so pixel-perfect in most cases, no matter the style (it was previously ok on windowsvista, with a bit of fuzzy padding, but on Fusion it was either too tall or short due to font height estimate stuff). the downloader and watcher lists, which dynamically size according to contents, also do so unerringly. I find this very slightly eerie tbh so I might add a couple pixels of whitespace on the end so your brain knows there isn't more to scroll. let's try it like this for a few weeks and see how it feels
  • added `--pause_network_traffic` launch parameter to the client, which hits `network->pause->all new network traffic` before the network engine initialises, to help debug when pending subscriptions or initial-session downloaders go crazy
  • in 'manage tag siblings/parents', if you import from a clipboard/txt that includes a `a->a` loop-pair, the dialog now yells at you
  • as a little hack, I added ctrl+shift+c to the taglist to say 'copy tags with parents' the default ctrl+c does not include the parents
  • removed a duplicate and blank entry for 'EXPERIMENTAL: Image path for thumbnail panel...' in `options->thumbnails`, whoops
  • I added a little safety code to the import folder, trying to catch a possible infinite loop we have seen. import folders now also take regular short breaks to give other threads like garbage collection a guaranteed moment to step in
  • fixed a bad buffer truncation calculation in the native video viewer

  • build/jpeg-xl stuff

  • in `help->about`, moved the Jpeg-XL and HEIF(/AVIF) support lines to the second tab, and together
  • if your client does not have Jpeg-XL support, the import error is now stated when you hit `help->about`. **if you are on macOS, you probably do not have Jpeg-XL support yet**, and the solution, whether on App or from source, is to run `brew install jpeg-xl` in the terminal, which is now also said in this message (issue #1670)
  • added a small note to the installing help docs to say Apple Silicon users should consider running from source. the App is Intel for the time being, and running from source will give you a much more pleasant experience all around

  • boring code cleanup

  • refactored the `ProcessContentUpdates` megacall and its friends to a new 'content updates' database module
  • refactored the duplicates file search stuff down to a new database module
  • and refactored the 'set duplicates' call down to a new database module
  • about 110KB of code is thus cleared out of the `ClientDB` monolith, which is down to 500KB, and we are ready to plug the duplicates auto-resolution module in as it now has scope on the various calls it needs
  • fleshed out the duplicates db module's master id definitions
  • cleaned up how I refer to file duration across the program. all the convenient-to-change duration or frame durations have a unified nomenclature and always say if they are in seconds or milliseconds

  • duplicates auto-resolution

  • the final difficult duplicates auto-resolution database problem (fast incremental duplicate pairs search) is solved. outside of at-scale testing and final integration, the database side of this is now complete. now just need to to figure out a daemon and a preview panel

• version 607

  • misc

  • jpeg-xl images are now fully supported! jpeg-xl is one of the contenders for 'what format do we use next?', and I think a pretty good one. it is basically jpeg but it can handle more colour depth/HDR, transparency, and optional lossless encoding, all while saving about 40% equivalent filesize to a jpeg/png. I understand it can also do animation (although we don't add that today), making it a potential 'capabilities superset' of gif as well. jpeg-xl is not well supported yet, but I hope that as more programs add support we'll see that change, and hydrus does its part today. thanks to the users who navigated this
  • added some boxes to `options->thumbnails` to give it some better layout and grouping
  • the 'show the media viewer top hover text in the status bar when one thumb is selected' option now defaults to True. it was a mistake to turn this off by default, and all users will be switched to True on update. the checkbox remains in `options->thumbnails` if you want to turn it off
  • added 'Give thumbnail ratings a flat background' to `options->thumbnails` to turn off if you prefer to just have the ratings without the rectangle backing I added last week
  • the new 'namespace grouping sort' sorted text list in `options->sort/collect` now has a paste button if you need to enter a ton of them
  • a new checkbox under `options->importing` let's you stop the program switching to the destination download page whenever you drag and drop a URL on the program. this was personally driving me nuts
  • when you right-click a selection of predicates that includes OR preds, the 'copy' submenu now includes an option to copy texts with collapse ORs, which renders them like "a OR b OR c" in one line, rather than the separate rows
  • the duplicate filter's "comparison statements" in the mid-right hover now show if one or both files have duration. I hacked this in simply for guys who are using external tools to get videos into the potential duplicate pairs queue, and I still want to completely rewrite this thing to a fully user-editable system once I have the tools matured in the duplicates auto-resolution work
  • a safety limit that would force the status bar to stick to 'x files' instead of the more interesting and cpu-intensive 'x images/collections/whatever' is raised from 1,000 files to 100,000. let's see how it goes
  • fixed a sometimes-error when showing different file presentations of watchers/gallery downloaders (when you right-click some and say `show files->blah`)
  • the complicated pre-download file status logic that determines if a URL-mapping has untrustworthy neighbours now prints a little log message of what it discovered when it hits a positive. this thing runs a little quiet and I want to see if it is firing too often or is not quite checking what we want in difficult cases
  • fixed a weird typo when unsetting the EXPERIMENTAL media viewer background bitmap thing in `options->thumbnails`

  • MPV

  • I did some work trying to handle the 'too many events queued' problems we've seen with mpv recently. I cleaned some code and reduced the overhead of my custom event handling, but I am generally convinced that some of the problems we've seen are due to 100% CPU bugs in various libmpv versions, particularly because I can set the 'loglevel' of mpv to 'no', for 'nothing at all', but the error still happens, suggesting there is probably no log-processing speed I can do on my end to fix this. I am leaving the emergency-dump-out code in place, but for this specific 'too many events queued' error it no longer adds that file to the 'never load again this boot' list--you can deselect the file and then reselect to try again (although it seems like the mpv player's event loop can be poisoned by the situation and lags a bit on next load). I have also seen the error occur at I believe the start of playback, but then I could load the file fine another time with the same dll, which suggests the log stuff can pile up due to a brief busy period in thread scheduling or whatever. so, there is more to do here--perhaps a 'if we get the error persisting for more than two seconds' or something
  • I did discover a different loop flag I could set for mpv that fixes one of the reliable problems--setting 'loop-playlist' rather than 'loop-file', which seems to force the file to reload and avoids the EOF rewind bug that mpv is running into. if you get the error on the first loop of a file, try hitting the new checkbox under `options->media playback` and we'll see how it all goes
  • I also just bit the bullet and added a DEBUG option, also to `options->media playback`, to disable the new 'too many events queued' dump-out hook. if you hit many false-positives and are feeling brave, try turning it off, but you may have to deal with some 100% CPU situations

  • viewtimes are now in ms

  • the filetime viewing stats system now records total viewtime in ms (previously it only had 'second' resolution). this doesn't matter all that much, but all the little deltas it adds are now 'viewed file for 3.2 seconds' rather than always rounding to the nearest integer
  • the settings under `options->file viewing statistics` are now full 'time delta' widgets that support ms, so you can say 'viewing a file for at least 2.5 seconds counts as a view' rather, again, than having to deal with full integers all the time
  • system:viewtime now supports milliseconds input for viewtime ranges
  • mr bones now reports a ms-precise (float) total viewtime (e.g. 3.456s) for the current search

  • client api

  • the client api can now see and edit file viewtime statistics!
  • `/get_files/file_metadata` now states the file viewtime statistics (last viewed time, total views, total viewtime for the different viewer types) for each file
  • file viewtime statistics has a new 'client api viewer' enum, separate from the existing 'media viewer' and 'preview viewer' record, if you'd like to do your own thing. I've hacked this a bit, and it doesn't show in the normal Client UI yet, nor can you search on it via `system:file viewing statistics`. the current options on how to show viewtime (`options->file viewing statistics`) are frankly crazy, so I think probably we'll want to go to a list of simpler checkboxes or something and stop trying to be clever
  • the new `/edit_times/increment_file_viewtime`, which needs the existing 'edit times' permission, lets you add views and viewtime and set the latest view timestamp
  • the new `/edit_times/set_file_viewtime`, which needs the existing 'edit times' permission, lets you set fixed values (i.e. if you are migrating from one client to another)
  • I have not added a delete/clear call here yet. if you would like to do this, let me know what you would like. the client only has 'clear everything for this file' and 'clear all records completely', but I expect we'd want the ability to clear by view type?
  • wrote help for this
  • wrote unit tests for this
  • client api version is now 78
  • the rich version of `/get_files/file_metadata` is now providing a lot of personal user metadata like inbox, ratings, local tags, and now viewtime statistics, making the 'just share some files with a friend' scenario less attractive. I was considering adding a new 'edit file viewtimes' permission today that would allow the edit and also dynamically show/hide the new viewtimes, but I think it would be better if I make a _read_ permission in the _ouvre_ of 'can see user file metadata', which, if absent, will mean callers can only ask for the basic file information. this permission may apply in other places, and the whole client api permissions system could do with a pass and likely some KISS. let me know what you think!

  • boring cleanup

  • cleaned up how some thumbnail-focus-delegation happens on new file focuses
  • cleaned some viewtime pipeline code generally
  • fixed the API help json example requests for `set_time`, which didn't have `file_id` stuff
  • replaced all my 'convert ms time to s time, but a float, oh except I guess when it is null, don't forget bro' code with a single and safer unified call across the program
  • cleaned up some agnostic 's or ms' duration calls, but there's plenty more to do

  • build updates

  • last week's 'future build' went well, so I am folding the changes into the normal build. users who run from source may like to run their `setup_venv` script again today, and users who would like to run from source but only have python 3.13 now have a route to run hydrus
  • details--
  • PySide6 (Qt) is updated from `6.6.3.1` to `6.7.3` (test version is now `6.8.1.1`, which source users on Python 3.13 can run)
  • on macOS, PyQt6 (Qt) is updated from `6.6.0` to `6.7.1`
  • OpenCV (image stuff) is updated from `4.8.1.78` to `4.10.0.84`, which lets us update numpy (test version is now `4.11.0.86`)
  • numpy is switched from `<2.0.0` to `>=2.0.0`. this adds Python 3.13 support to hydrus for source users
  • pillow-jxl-plugin is added, and thereby we have Jpeg-XL support (thanks to some users for navigating this!)
  • the python mpv package is updated from `1.0.6` to `1.0.7` (test verison stays at `1.0.7`; there is nothing new)
  • twisted (the networking engine that runs the hydrus server and client api) now includes better TLS and http2 support
  • some import hacks that helped old PyInstaller navigate numpy and OpenCV bundling are removed

• version 606

  • tag sort

  • when you group sorted tags by namespace, you can now force the order of the namespaces! this means you can have all the creator tags first, then the series, then the character, etc..., in any order you want. specifically, tag sorts in 'tag' or 'count' mode have a new 'group by namespace (user)' option that applies the sort override. you can set the namespaces you prefer under a new list in `options->sort/collect`, and the default is `[ creator, series, character, species, unnamespaced, meta ]`. any namespaces that do not fit will be grouped (a-z) underneath, just as before
  • sorry for taking so long to get to this. thankfully it worked out well, enough that I have set 'group by namespace (user)' the default for new clients; I recommend everyone who is familiar with normal booru namespace sort set this too under their `sort/collect` page
  • tag sort will handle some unusual unicode character comparisons (e.g. 'ß' vs 'ss') better
  • tag sort will now reliably sort non-ascii namespaces above unnamespaced tags when grouping by namespace

  • misc

  • the `database->view file history` chart now has checkboxes for all four lines. its code is cleaner and it now updates itself faster and nicer, and charts from old searches are now deleted promptly
  • fixed the position of the 'collection' thumbnail icon when you have the new 'show ratings on thumbs' set. also, the backing colour on the collection count now covers the icon; the colour of the backing panel and texts now feed off your current stylesheet; and I generally cleaned up how the icon and text position themselves
  • after last week's not-excellent background highlighting for ratings on thumbnails, I bit the bullet and just did a flat background with your normal qss window panel colour. inc/decs get a background too. it looks _fine_ and thumbnail ratings are now clear in all situations--it basically looks like the top-right hover now
  • the file right-click menu's top row flyout metadata submenu, which shows details like file modified time, now includes the exact file size in bytes
  • the top hover window's 'EXIF and other stuff' button is moved to the center button row, has a new 'page with text' icon, and is always available since it now will always show the contents of the flyout metadata submenu. you can thus now see the data in this menu from the archive/delete or duplicate filters

  • boring code cleanup

  • in bad-idea-cleanup twelve years in the making, refactored the 'listening media list' out of the navigable canvas subclasses. the underlying list is now handled inside the object rather than being the UI panel itself
  • the duplicate filter also gets some 'listening media list' cleanup. it handles its own content updates and the handling of what to do after deleting a file in the pair is now more safely wrapped inside the same atomic event
  • updated up an old 'remove media' pubsub, from the browser media viewer to the underlying thumbnail grid, to be a nicer Qt signal
  • updated the same thing from the archive/delete filter's commit, which can have special remove logic when the filter is complete, and replaced a hacky second-remove, which tries to catch users who hit F5 very quickly after an archive/delete is complete, with a popup that appears after two seconds to show slow commits actually happening--we'll see how it does IRL
  • fixed some recent 'woah that text-and-thing went to the right' bad layout in the 'edit shortcuts set' and 'edit subscription' panel--thank you for the reports
  • optimised some media result load
  • cleaned up some media result caching sync around the database repair code
  • deleted some static old colour defs that I missed in previous sweeps. now I updated the colours in the thumbnail collection stuff, none of them are used any more
  • reworked more garbage thumbnail ratings layout code
  • did some tag sorting KISS refactoring

  • duplicates auto-resolution

  • broke the new database module into two--one for the clever search side, the other for the simple storage side. the main (duplicates) db module will now see the storage and keep it updated on new/dissolved potential duplicate pairs
  • merged the rule-setting gubbins together into one 'set rules' command and added search-resetting code on search updates
  • wrote a status-count cache for quick review of rule progress
  • brushed up maintenance code for orphan rules and pairs
  • fleshed out the 'let's see if these unsearched pairs match our search' tech and the 'let's see if these search-matching pairs pass our auto-dupe rules, and if so, set the action' tech
  • refactored the media results generation and caching code from the monolithic `ClientDB` to a new module. the duplicates auto-resolution search module now talks to this guy
  • I still have to refactor a couple more things so I can wire this all up, but this should be simple work. all the difficult parts of the duplicates auto-resolution db stuff, except for one bit of incremental search I need to figure out, feel like they are fixed. only one big difficult hurdle (the rule preview UI) remaining!

  • future build

  • I am making a 'future' test build today--it should be in a post beside the normal build. it is the same as the normal build except it has jpeg-xl support and a number of libraries are updated to new versions. I would like advanced users to try it out and give me feedback on any boot problems. instructions are in the release post
  • build details--
  • PySide6 (Qt) is updated from `6.6.3.1` to `6.7.3` (test version is now `6.8.1.1`, which source users on Python 3.13 can run)
  • on macOS, PyQt6 (Qt) is updated from `6.6.0` to `6.7.1`
  • OpenCV (image stuff) is updated from `4.8.1.78` to `4.10.0.84`, which lets us update numpy (test version is now `4.11.0.86`)
  • numpy is switched from `<2.0.0` to `>=2.0.0`. this adds Python 3.13 support to hydrus for source users
  • pillow-jxl-plugin is added, and thereby we have Jpeg-XL support (thanks to some users for navigating this!)
  • the python mpv package is updated from `1.0.6` to `1.0.7` (test verison stays at `1.0.7`; there is nothing new)
  • twisted (the networking engine that runs the hydrus server and client api) now includes better TLS and http2 support
  • some import hacks that helped old PyInstaller navigate numpy and OpenCV bundling are removed

• version 605

  • ratings on thumbnails

  • thanks to a user, ratings are now displayable in thumbnails! hit the new 'show in thumbnails' service checkbox in _services->manage services_ and that service's ratings will show
  • I updated this a bit and added a second option for 'show even when there is no rating value'
  • I played around with different backing colours to make these new ratings stand out more. in the end, my best solution was very hacky and isn't amazing--the stars in particular can get washed by a banner or busy thumb underneath. I tried a block of backing colour but it all looked worse. we may be approaching a 'just do it with svgs' moment for ratings rather than my decade-old painter primitives and coordinate lists. have a play with it and let me know what you think
  • I improved some rating drawing positioning, too--some stuff was off by half a pixel etc..
  • thumbnails now refresh when after a _manage services_ ok

  • misc

  • did a hotfix a couple hours after v604 release, v604a, to fix an issue with double/middle-clicking collection thumbnails to launch the media viewer
  • I deleted the Endchan /hydrus/ board and removed the links from the client and help. the board was intended as a bunker and never got much traffic, but the whole site being spammed this week reminded me that I don't want to own a board any more. if we ever need a bunker again, I'll revisit the issue
  • fixed some file-picker dialogs' name filters, which were not filtering to PNG or JSON correctly. all file dialogs that filter files this way now also offer `Any files (*)` as a second option
  • the Lain import downloaders dialog now filters the file-picker to .png files
  • I reworked the 'top hover file summary' settings under `options->media viewer` to their own box with a bit of extra text, and I renamed the complementary setting under `options->thumbnails` to the clearer "Show the media viewer's top hover file text in the status bar when a single thumbnail is selected"
  • I did a bunch of layout work this week, cleaning up how things position and expand to fill available screen space. there were several hundred things impacted and I did not have time to check everything that might have been changed. please let me know if some dialog has a help button that's weirdly aligned etc.., thank you!
  • cleaned up the reason-initialisation of the advanced file deletion dialog. the list of reasons is now much more item-position-stable and will not create duplicates when intersecting options clash. the selection rules are now simple: if all the file(s) have existing deletion reasons, this reason is listed, marked as the existing reason, and selected; otherwise, if the dialog is set to remember the previous deletion reason, this is listed and selected; otherwise, the "default" reason the dialog launches with is listed and selected. items have a more stable position as the list now always follows an order of (optional unique default shared reason, list of pre-defined reasons, optional unique shared existing reason, optional do-not-alter existing reason, custom reason), and items are marked in-place if they are interesting (issue #1653)
  • fixed 'do not verify https' network jobs for clients with a CA bundle set in their envs

  • duplicates

  • in a terminology change that matters in other places, the duplicate filter, for the current file index, no longer says 'A' or 'B': but now 'File One/File Two'
  • the comparison scores list in the mid-right duplicates hover window now says the total score difference as an actual number. let's see if the IRL scores are helpful as we move into making these rules more user-customisable

  • default downloaders

  • the e621 site has 'contributor' tags now, to distinguish VA talent or model makers from the primary artist. I was going to fold this into the normal default e621 parser as just another 'creator' tag, but the examples I found were pretty spammy so I'm not sure it is so useful, at least out of the box for normal users. if you are super interested in this, you might like to check out the new 'e621 file page parser with contributor tags' under `network->downloader components->manage url class links` for the `e621 file page` url class. the tags all seem to have `_(va)` kind of thing after them, so they wouldn't _confuse_ our existing creator tags, but it does seem like a lot of incidental spam and maybe it muddies things and should indeed be its own namespace in hydrus or just be ignored, idk, let me know what you think

  • style

  • thanks to a user, the e621 stylesheet gets some tweaks and better darkmode colours. check the new 'e621_redux' QSS. there's some some interesting new transparency tech on taglists

  • client api

  • fixed an issue with the `/manage_file_relationships/set_file_relationships` call and the 'set B as better' duplicate action (seen at times in other areas of the client, very rarely, as "set as worse" or "this is a worse duplicate") with `do_default_content_merge`--it was doing nothing, since there is no default for this action. now it fetches the normal 'set A as better' default options (also making sure to apply them the correct way around ha ha)
  • client api version is now 77

  • boring code cleanup

  • across the duplicates system, I've reworked a tangle of references to 'first' and 'second' or '1' and '2' to a unified 'A' and 'B' for our pair processing. in code and duplicates action settings, the A is the first file in the pair being actioned and B is the second. in a few places where we have yet determined AB, I now specifically use 1/2. it is arbitrary, but at least it is now clear
  • refactored the 'edit duplicate content merge options' panel to its own file and converted it to a non-scrolling widget so I can embed it in panels easier. it can also take SetValue calls after init and will reconfigure itself for different duplicate action types
  • did some more Qt Enum updates since my linter found a bunch more: `QPainter.RenderHint`, `QPainter.CompositionMode`, `QPalette.ColorRole`, `QColor.NameFormat`, `QFont.Weight`, `QFont.StyleHint`, `QFont.StyleStrategy`, `QImage.Format`, `QTextCursor.MoveOperation`, `QColorSpace.NamedColorSpace`, `QTextOption.WrapMode`
  • boring layout cleanup
  • I did some Qt research and fixed a jank expanding layout technique I use in ~90 different places. a bunch of panels should eat up extra pixels a little more reliably, with fewer cases of the invididual widgets all experiencing cosmological redshift or mysterious magical growing lower buffer space if the dialog grows
  • also fixed a bunch of bad sizer flags and stretch stuff used on my text&widget gridsizer that I use in ~190 places
  • fixed some crazy flags and layout used inside my text&widget gridsizer
  • fixed up some crazy layout in manage siblings
  • fixed some crazy popup toaster layout
  • fixed crazy layout in the login credentials edit panel
  • fixed some crazy layout in the shortcut action editing UI
  • fixed some crazy layout in the new duplicate pairs search context panel. it now expands to fill available space properly
  • fixed several options pages that didn't know what to do with extra space
  • the system:hash panel list now expands vertically
  • reworked the view options dialog under the `options->media playback` file list into one clean and lined-up gridbox

  • duplicates auto-resolution

  • finished the 'action' tab, which governs the duplicate type to apply to the pair, whether to delete A or B, and any custom duplicate content merge options
  • gave the rule dialog workflow a slight pass, particularly making the 'comparator' step define 'A' and 'B' rather than 'better' and 'worse'. my head was too into setting better/worse duplicates, which made things things awkward for same quality, alternates, or false positive actions. similarly, the 'action' step is now orienting towards position-specific 'A is better' verbs rather than 'yeah I guess set the better as better bro'. also, as before, as soon as this panel was finished, I immediately disabled it for the first test, ha ha
  • the one-file comparator can now do 'either A or B has property x' tests
  • added some safety code to the rule dialog
  • work still to do is: the master database search and caching code, the preview panel, which will load up some example pairs and show them in a nice way (⊙ _ ⊙ ), tying all the objects finally together and saving them to db, and then the daemon that'll work rules on demand and in the background. I feel ok about most of it, but the db stuff may be a nightmare, and the preview code, which will need thumbnails and media viewer tech from a dialog to be worth something, will definitely be one. it'll be great to finally have 'thumbs anywhere' tech though, so lfg

• version 604

  • misc

  • fixed an odd and fairly recent bug where if you had an mpv window loaded in the preview panel--but it was paused--and then you changed pages back and forth, the mpv would not re-initialise itself with the previously paused media (nor really any media) unless you did some weird stuff where you clicked on another media and then changed pages again
  • system:filetype now supports exclusion (i.e. `system:filetype is not video`). also, the search system now allows multiple system:filetype predicates--it figures out the correct intersection (previously it just pseudorandomly selected one of them). the system predicate text parser can handle the new 'is not' operator, and I added a unit test to make sure it sticks
  • if you paste hashes with hash_type prefixes (e.g. `md5:775858135a6006db32f8ef0e5fa3102c`) into 'system:hash', it now auto-strips the prefix and sets the choice for you
  • the new 'fix missing file archived times' dialog pre-job now counts up and says the number of missing timestamps of each type it discovers. this thing seems to run at 100,000s of rows per second, so it turns out no worries
  • export folders should run a little bit faster now, especially to disks with higher latency (HDD or NAS, rather than a local SSD)
  • export folders now only do their sidecar exports if the respective file was actually copied this run (previously it looks like they were updating to the latest data on every run, aieeeeee!)
  • I've reworked the routine that draws the hover window into the background of the media viewer to more natively emulate out its layout and margin/spacer/etc.. sizes. the notes position still has some trouble, especially with difficult unicode, but it is better and now has justified text that properly expands to fill all the available space. this is now generally pixel perfect on my dev machine. while I am confident _some_ of it will survive janked QSS styles, some may not, so let me know what you see
  • the tags list in the background is now also pixel-perfect-positioned with its hover!
  • fixed the 'incdec' rating widget looking fuzzy in some places
  • the Edit Tag Filter Panel text inputs (and their paste buttons) are now add-only. previously, it would silently do add/remove logic on each item depending on whether it was already in the list, which is not helpful now we have big lists here. so, you can paste a big list of a hundred things and now it won't remove any stuff that was already in there
  • added `show file trash times/reason in top hover summary` checkboxes to `options->media viewer`, which expands the recent top hover summary stuff (issue #1652)
  • rejiggered some layout in the duplicates filter page sidebar panel. the search is its own box now, as is the options

  • sidecars

  • the import folder system now moves all possible sidecars when you have it set to 'move the source file'. it was previously still on the old system of just supporting a single 'filename.txt' sidecar format--now it does anything you have set up, live
  • the import folder UI has a couple new tooltips saying 'hey bro, delete and move actions will delete/move sidecars too, so if you have one sidecar for multiple files, make sure not to set the action to delete or move, or the next file will not have a sidecar to see'
  • hooked up/improved the edit sidecar panel test UI: the string processor button on the main sidecar 'metadata router' panel now loads up example strings from your actual importers and file paths; the string processor button on the edit single sidecar importer panel now loads up example strings from your actual file paths; the process that loads up expected result strings for the multi-column test panel is more tolerant to failure; the process that provides example test data to the sidecar importer panel is more tolerant to failure; and the importer panel's test no longer spams JSON garbage into the test context, what was I thinking. I also just cleaned some of this code

  • downloaders and network

  • thanks to a user, the e621 login script is fixed! if you have had errors with this recently, then the update _should_ just reset any error state and allow it to try again immediately, but if not, check `network->logins->manage logins` and make sure the `active` is set and hit `scrub invalidity/delays` if you need to (issue #1655)
  • updated the simple-tag shimmie file page parser to not grab 'popular tags' when those appear
  • I am removing the old deviant art file page parsers from the defaults. we removed the search a while ago, but there was limited drag and drop support until seemingly just recently. it looks like the whole site got a revamp. I had a very quick look at their new format, and while it may be possible to write a new parser, I will leave this to the user-run repo to handle if they wish. I understand the whole site is mostly wrapped up in OAuth stuff now, so I think imgbrd-grabber is probably a much better solution here going forward
  • the 'review bandwidth' panel gets a pass: its multi-column list ctrl is now wrapped in a nicer panel that has live buttons for edit/delete; the 'edit defailt rules' button now lists the 'global' rules; there's a little help label; and the 'search distance' column now shows your all-time total bandwidth use if you have 'show all' set. the individual network context bandwidth review panel (the one with the bar chart) also now shows your all-time usage. let's see what our all-time globals are lol
  • brushed up the 'getting started with downloaders/subscriptions' help a little bit
  • also expanded the 'bandwidth' section of this help and added a couple screenshots

  • old mpv dll

  • I am rolling back the Windows build's recent mpv update, back to the 2023-08 mpv dll. the 2024-10 dll caused stutter and slowdown with several users in the wider test, particularly Windows 10 guys. we'll revisit the topic in a future update round and see if we can select a better test candidate
  • for those source users who are still on the new dll (myself included), if your client does run into an APNG or webm/mkv that trips 100% CPU (it is some bananas silent logspam outputting too fast to process), this is now recognised after a couple seconds in the hydrus error handling and the file should unload itself with the "Sorry, this media appears to have a serious problem!" error dump-out

  • boring technical stuff

  • the archived-file delete-lock now plays with normal trash maintenance better. the 'do we have any trash files to delete?' method now works more flexibly and it can find stuff to delete even if the oldest 256 items are locked (previously it couldn't!)
  • on db update, everyone is going to get a file maintenance job queued up to check their images are all in the similar files system as desired. a recent bug caused some files not to re-register with the similar files system if they were re-imports. this job isn't super heavy on CPU or anything, so no worries, it'll just do its thing in the background and maybe fix up some holes
  • under `file->shortcuts`, the custom shortcuts list now forces unique names on adds and edits
  • the 'network report mode' in the `help->debug` menu now spams the Request and Response headers of each job. I also clarified, made error-safe, and added newlines to some of the other popups here to make things clearer when URLs are being transformed and so on
  • the `setup_desktop.sh` script now wraps the Exec line value in quotes, so this should now work if your hydrus install folder includes whitespace

  • boring code cleanup

  • harmonised my various daemons more into the parent `ManagerWithMainLoop` class, which now takes responsibility for the mainloop status tracking and pre-loop startup wait. the way these guys now do their pre-work wait is a little smarter and will react to early program shutdown and so on more cleanly and in the same reliable way
  • moved more old multi-column list calls to the newer methods that do cleaner post-edit select, scroll, and sort. this includes some stuff in parsing UI and manage services, so hopefully these lists will be just nicer to work with
  • made the newer multi-column list code even simpler when just adding one item
  • fixed up some more static superclass calls to `super()` in the autocomplete code
  • merged `CanvasWithDetails` and `CanvasWithHovers` so I'd be able to interrogate the hovers when figuring out the details layout gubbins
  • moved the 'open the media viewer from the preview panel' action from my old pubsub to a nicer Qt signal
  • fixed some crazy sizing in my 'select stuff from a list' quick dialog

  • duplicates auto-resolution

  • updated the Metadata Conditional objects to work on full file search contexts instead of single predicates
  • added inbox and archive support to the Metadata Conditional system
  • wrote edit UI for my 'pair selector/comparator' objects
  • refactored and decoupled my autocomplete dropdown classes in a couple little ways and wrote a new stripped-down autocomplete dropdown for the metadata conditional edit UI
  • wrote that first 'Edit Metadata Conditional' panel
  • the first version of the 'comparison' tab of the new duplicates auto-resolution UI is thus complete, and with that, I immediately disabled it like the 'search' tab, hahaha

• version 603

  • misc

  • fixed a typo that caused the 'sort files by' menu to (ironically) sort by crazy means
  • fixed a bug with the time delta widget where the ms would not set to 0 when it initialised with a value that has no milliseconds component but the minimum allowed value had a milliseconds component
  • the 'force metadata refetch' thumbnail submenu now shows actions for just the focused file, and in the media viewer it now shows these actions for the current file (previously this submenu was accidentally a stub in the media viewer since there is no concept of a 'multi-file selection' up there)
  • the 'review bandwidth usage' panel now initialises its widgets immediately after it opens, not half a second later
  • added some safety code to ensure file re-imports (and probably some other weird file import situations) integrate correctly into the similar files system
  • added some EXPERIMENTAL options just for me to `options->speed and memory`

  • archived file delete lock

  • I have had a think about the delete lock in hydrus. I have never liked this system because it interacts with some complicated file service logic and inserts awkward logical workflow exceptions. furthermore, my initial implementation has not played well with multiple local file services. it was also imperfect, since certain signals to 'delete from all local services' or some odd variants of 'delete from trash' could skip the lock, and it disallowed removal of a file from one local file service even when the file was in others. several users have asked for various exceptions, either on an ad-hoc basis of for duplicate filtering. I played around with trying to fix and implement some of this stuff this week and realised I was digging an even worse hole for myself. I have decided to KISS and scale back what the lock does down to the simple emergency case of not wanting to lose nice things. **therefore: the archive-delete lock, henceforth, will only test for deleting files from the trash, i.e. a physical delete**
  • the option UI is updated to say this, and I cleaned up and updated a bunch of hellish hacky code all around here
  • the normal manual delete files dialog now filters the 'delete physically' and 'delete physically/no record' options according to the delete-lock, no moaning or popups
  • this is obviously a workflow switch, and I apologise for the inconvenience. I know the guys who care about this do care about it. I should not have tried to make it complicated in the first place. let me know what works and what doesn't, but I will insist on keeping this whole thing KISS going forward. if you use the trash for storage, please consider making a new local file service and putting your unusual 'maybe I'll delete it' files there

  • trash deletion rules

  • while I was poking around the archived file delete lock stuff above, I saw some hacky delete logic. in several semi-automatic systems I saw code that would say 'send the file to trash, unless it is already in the trash, in which case "upgrade" to physically delete it'. I am making the formal choice to no longer do this, and this code is now amended to say just 'send the file to trash if it isn't already there. specifically--
  • the duplicates filter page will no longer allow you to set a search domain in 'trash' or 'all local files' or 'all deleted files' (or, for advanced users, 'repository updates' lol). if it is somehow given a trashed file to process and receives a duplicate action that includes a file delete (like 'this is better, and delete the other'), it no longer tries to physically delete it; it just leaves it in the trash
  • the Client API `/manage_file_relationships/set_file_relationships` call is the same; when you say to `delete_a/b`, it'll now ensure the file goes to the trash and that's it
  • the archive/delete filter has long not allowed trashed files, but it too now no longer has any special logic for trashed files that it happens to encounter (e.g. the file is trashed by other means after the filter is created); it will now never provide a 'delete from hard disk' option in the final commit dialog. the top 'delete from' item in the commit choice, which is usually the current location context, is also filtered and selected more carefully for users who use multi-location search domains
  • the manual export and export folders now explicitly, when set to delete files, now send to trash; they never delete from the trash
  • I expect I've missed some clever situation, but typically, now, files are going to be physically deleted only if the `options->files and trash` settings kick in or you the user force it manually, and of course the new archived-file delete-lock prohibits this final step no matter the source

  • media viewer

  • fixed a bug where the top-right hover window was, when the mouse is over it and the media changes from one with no URLs to one with URLs, not be able to immediately figure out how tall the URLs list should be and was giving you a height-truncated window
  • I also, after some head banging and dark art, seem to have finally and properly fixed the annoying adjust-flicker that can happen to top-right hovers with urls or note hovers where a moment after showing it may grow three pixels taller etc... the top-right and center-right hovers now appear to size perfectly every single time; at least on Windows, I cannot break them even if I try. to keep things clean, I have removed some old hacks we built up over the years, but perhaps some of these are still relevent, so let me know how things appear on different OSes
  • I improved the (re)layout after you hide/show a note with middle/right-click. it _should_ recalculate its new size immediately
  • the notes that are drawn in the background of the media viewer are now always as wide as the hover window that pops up over them. I improved the padding calculations, so they are more closely aligned with the hover, but it isn't perfect yet--however I think I will be able to make it so in future
  • fixed a variety of issues with the media viewer volume button and its slider flyout. it could stay open on media change and in some cases open up if the mouse was over where the flyout should be on a media change, or flickering into just slightly the wrong overlapping location if the mouse comes in at the wrong angle. there's still a couple weird ways you can break it (e.g. sending a 'move media' keyboard shortcut while the mouse is over the slider), but I'll leave that for another day
  • fixed an issue with the media viewer's top hover window disappearing while your mouse was over the volume slider
  • reduced some change-media flicker with the seek bar in the media viewer

  • misc cleanup

  • I replaced a bunch of `isVisible` with `not isHidden` in Qt. I was using the former for some widget and panel hide/show situations, but they are not quite the same: `isVisible` tests up the ancestor heirarchy; `isHidden` tests only the given widget's visibility bool
  • reworded the 'blacklist' explanation a bit in 'getting started with downloaders' and added a screenshot
  • updated the Linux running from source help to talk about `libgthread`, the lack of which may stop Qt6 from booting
  • I cleaned up the `/manage_file_relationships/set_relationships` Client API call a little, including fixing a slightly incorrect object type that was missed in a recent rewrite of the duplicates content pipeline. I am not sure if this was causing any bugs, but it is better now
  • brushed up some of the labels/tooltips in the `options->file viewing statistics` page (issue #1644)
  • stopped the logging of a 'custom' `REQUESTS_CA_BUNDLE` with the new `options->connection` debug checkbox if it is what we would have set anyway (this was occuring if you went `file->restart`, since the new instance shares the same process/env)
  • fixed some unit tests for the new duplicate merge delete rules
  • did some misc linting

  • fixed up tag filter UI

  • everything is in layout boxes now, some collapsible
  • fixed up some layout flags so things are aligned or expand better, and the global namespaces list shouldn't have a scrollbar any more

  • macOS build

  • updated the macOS build script to retry the hdiutil dmg-building step multiple times in the very frequent case of it, seemingly, getting lock-kekked by XProtectBehaviorService (https://github.com/actions/runner-images/issues/7522)

• version 602

  • media viewer top hover file info line

  • added four checkboxes to `options->media viewer` to alter what shows in the media viewer top hover window's file info summary line. you can say whether archived status is mentioned; and if it is, if there should be a timestamp; and you can say whether individual file services should be enumerated; and if so, whether they should have timestamps
  • this same line is reflected in the main gui status bar when you have the new `options->thumbnails` checkbox set on--so if you missed seeing your current local file services on the status bar, it should be doable now, and in a more compact way

  • archived times

  • there has been a bug for some time where if you import files with 'automatically archive' set in the file import options, an 'archive time' was not being recorded! this is now fixed going forward
  • for the past instances of this happening, I have written a new `database->file maintenance->fix missing file archived times` job to fill these gaps with synthetic values. it can also fill in gaps from before archive times were recorded (v474, in 2022-02)
  • on the v602 update, your client will scan for this. **if you have a large database, the update may take a couple minutes this week**. if you have either problem, it will point you to the new job
  • the job itself looks for both problems and gives you the choice of what to fix. for files imported since 2022-02, it assumes these were 'archive on import' files and inserts an archive time the same as the import time. for files imported before 2022-02, the synthetic values will be 20% between the import time and (any known deletion time or 2022-02), which is imperfect but I think it'll do

  • system predicates

  • the system tags for `duration`, `framerate`, `frames`, `width`, `height`, `notes`, `urls`, and `words` are now written `system: has/no xxxxx` rather than `system:xxxxx: has/no xxxxx`
  • the `system:file properties` UI panel is now a two-column grid. 'has xxxxx' on the left, 'no xxxxx' on the right
  • `system:has/no duration` added to `system:file properties` (it is also still in `system:duration`, but let's see how it goes having it in both places. I am 50% sold on the idea)
  • the 'paste image!' button in the `system:similar files` edit panel now understands file paths when they are copied from something like Windows Explorer. previously it was only reading bitmaps or raw text, but when you hit ctrl+c on an actual file, it actually gets encoded as a URI, which is slightly different to raw text. should work now!
  • fixed the `system:duration` predicate's string presentation around 0ms, where it would sometimes unhelpfully insert '+/- no duration' and other such weirdness
  • fixed the `system:duration` edit predicate window initialising with sub-second values. previously, the ms amount on the main value or the +/- were being zeroed
  • the various `NumberTest` objects across the program that have a +/- in absolute or percentage terms now test that boundary in an inclusive manner. previously it was doing `x < n < y`; now it does `x <= n <= y`, which satisfies 5 = 5 +/- 0, 6 = 4 +/- 50%, and 0 = 600 +/- 100%
  • the 'has/no' system pred shortcuts now all parse in the system predicate parser. the old format will still parse too. I added unit tests to check this, and more to fill in gaps for 'framerate' and 'num frames'

  • rating predicates specifically

  • system:rating search predicates are now edited separately, which means that in the edit panel, each rating widget has its own 'ok' button. you edit one at a time with a clearer workflow
  • the radio buttons involved here are now less confusing. no 'do not search' nonsense or 'is = NULL' to search for 'no rating'--you now just say 'has rating', 'no rating', or set the specific rating
  • the inc/dec system pred box now has quick-select choices for 'has count' and 'no count'
  • rating predicates have slightly simpler and nicer natural language labels. 'is like' and 'more than 2/3' rather than '= like' and '> 2/3'. 'count' instead of 'rating' for the inc/dec ratings. inc/dec rating predicates will also now swap in the new 'has count' and 'no count' too, rather than saying "> 0" explicitly
  • the system predicate parser is updated to handle the new 'count' labels (but the old format will still work, so nothing should break™). new unit tests check this
  • these panels now recover better from a predicate that refers to a service that no longer exists. in general, if you try to edit a bad pred (e.g. from an old session) it'll try to just ignore it and give you a popup letting you know

  • deleted similar files search

  • the similar files system no longer de-lists files from the search tree when they are deleted. phashes being disassociated from deleted files was not intentionally enforced before, but it was legacy policy from an old optimisation when the search tech was far more limited than today; now the file filtering tech is better, and we can handle it CPU wise, and I now intentionally want to keep them. it should be possible to search similar deleted files in future. the similar files search code can get pretty wew mode though, so I wouldn't be surprised if there are some bugs in odd 'all known files' situations
  • I do not think this functionality is _too_ useful, but we might want to build a 'oh we have seen this file before and we deleted it then, so stop this import' auto-tech one day, maybe, if we combine it with multiple/better similar files hashes, so keeping deleted file data in our search trees will be the policy going forward. although I regret these perceptual hash disassociations, the good news is that pixel hashes were never removed, and this will be more useful for this objective

  • misc

  • renamed 'human-readable embedded metadata' to 'embedded metadata' across the program, mostly just to stop it being so wide. I hate this system predicate, and with the jpeg stuff it is now a catch-all and basically useless as a discriminator. I have plans to replace it with a flexible all-in-one system that will integrate all the 'has icc profile' stuff together and bundle in fine search for 'jpeg:interlaced' or arbitrary EXIF data row search so you can search for what you actually want rather than the blanket 'uh some metadata I guess'
  • the 'move files now' dialog in `database->move media files` has a new 'run for custom time' button so you don't have to do 10/30/60/indefinite. note I also hate this whole system and want to replace it in the middle-term future with a background migration job, so fingers crossed this whole mess will be gone before long
  • when you manually lock the database with the Client API, the bottom-right status bar cell now says 'db locked'. when it is reading or writing, it now also just says 'db reading/writing' rather than 'db read/write locked'
  • the way the database updates the status bar with 'db reading' and stuff is now a little overhead efficient (it only updates the db cell, not the whole status bar)
  • clarified the text in `options->importing`
  • moved the 'default export directory' option from 'files and trash' to the new 'exporting' panel
  • if an SSLCertVerificationError occurs in a connection, the exception now has some extra info explaining the potential causes of the problem (issue #1642)

  • env stuff

  • I added a DEBUG checkbox to `options->connection` that explicitly sets the `REQUESTS_CA_BUNDLE` environment variable to your `certifi`'s `cacert.pem` path, assuming that path exists and the env variable has not already been set. I'd like to test this a bit and see if it helps or breaks any situations, and maybe force it one day, or add some more options. also, I don't know much about `CURL_CA_BUNDLE`, but if you want me to defer to that rather than `certifi` if it is already set, or you have other thoughts, let me know what you think
  • added `help->debug->data actions->show env`, which simply spams it to a popup and writes it to the log. it also splits up your various PATH vars for readability, but the list is usually giant so this is best actually read in the log rather than the popup atm. a couple other places where the env is spammed to screen also now uses this now format

  • Qt enum cleanup

  • fixed up more Qt5 Enums to the new Qt6 locations, including those now under `Qt.ItemDataRole`, `Qt.ItemFlag`, `Qt.SortOrder`, `Qt.ContextMenuPolicy`, `Qt.Key`, `Qt.KeyboardModifier`, `Qt.MouseButton`, `Qt.DropAction`, `Qt.AlignmentFlag`, `Qt.LayoutDirection`, `Qt.Orientation`, `Qt.CursorShape`, `Qt.WidgetAttribute`, `Qt.WindowState`, `Qt.WindowType`, `Qt.GlobalColor`, `Qt.FocusReason`, `Qt.FocusPolicy`, `Qt.CheckState`, `Qt.TextElideMode`, `Qt.TextFormat`, `Qt.TextFlag`, `Qt.TextInteractionFlag`, `Qt.ShortcutContext`, `Qt.ScrollBarPolicy`, `Qt.TimerType`, `Qt.ToolButtonsTyle`, `Qt.PenStyle`, and `Qt.BrushStyle`
  • and `QEvent.Type`
  • and `QItemSelectionModel.SelectionFlag`

  • other code cleanup

  • cleaned up the recent 'make width sort before height in most places' hacks into something nicer
  • deleted the old 'Edit Multiple Predicates' panel py file, which was only handling the ratings stuff. was probably an interesting idea at some point, but it was too convoluted IRL, both for users and me to work with
  • decoupled some rating-to-stars and stars-to-rating rating conversion code out of the rating services object

  • new macOS App

  • sorry for no macOS App last week; github retired the old 'runner' that builds the App, and I missed the notifications. we are today updating from `macos-12` to `macos-13`, which appears to still work on intel machines
  • `macos-14`, whose migration will presumably come in a few years, will likely require Apple Silicon (ARM), at which point I'll have to tell older mac users to run from source, but that's a problem for the future

• version 601

  • this page is still importing

  • when you try to close the client or a page of pages and one of the sub-pages protests with a reason like "I am still importing", you now get a yes/no dialog with an extra 'no, but show me the pages' button that will spawn a window listing buttons for every page that protested. clicking a button takes you to that page. this window is a frame, not a dialog, and will not go away on a click. if a page is subsequently closed, clicking the button greys it out

  • misc

  • the 'archived time' pretty text string is no longer flagged as an 'uninteresting' line, which again, as intended, elevates it to the top hover window and main gui status bar if you have detailed info set to show
  • `system:width` is now before `system:height` in the `system:dimensions` flesh-out panel. I also hacked this in the 'edit multiple preds' panel for existing fleshed-out predicates, but it is a whack implementation like I did for the sort stuff last week. as I've discussed with some others, the real answer here is probably a `system:resolution` that combines the two
  • re-classified the 'move flag' DnD option under `options->exporting` as a BUGFIX option, and altered the tooltip
  • fixed the new `Set to "all my files" when hitting "Open files in a new duplicates filter page"` `options->duplicates` checkbox, which was not saving on dialog ok
  • changed the 'needs work' tab name suffix on the duplicates filter page to 'x% done'. it will max out at 99.9% and then hide, never rounding up to 100.0%
  • added `Hide the "x% done" notification on preparation tab when >99% searched:` to `options->duplicates` for those who always want to see this for any outstanding work

  • sidecar importers

  • multiline .txt note sidecar importing is fixed. I previously added some 'clear out empty lines' parsing input cleaning, but this collapsed multiline content by accident. this content-agnostic stage of import is not where cleanup should occur!
  • some explicit unit tests now test CRLF splitting and multiline note parsing from .txt files (previously it was just doing tags). multiline note sidecars have broken a couple times now, precisely because it was un-tested. it will not break for so stupid a reason again!
  • fixed some stupid scrollbars appearing on the 'destination' panel of the main 'metadata migration router' (sidecar job) edit panel, and made sure the 'note name' text input can't get super thin (issue #1634)
  • when a multi-column list is given multi-line content for a cell, it now says `[top line]... (+n lines)` so you know there was more (previously it just trancated to the top line). this now pops up in a couple of note parsing test panel places
  • notes are now specified as notes in the main sidecar path list with expected content (they looked a bit like tags before)
  • it isn't a big deal, but a thing that sorts the sidecar-imported text rows before handing them off to the exporter now only does a (namespace-aware) tag-sort if the exporter is tags, and otherwise just does a straight-up normal text sort

  • some dialog validation

  • the 'edit cookie' panel now strips leading and trailing whitespace from the name, value, domain, and path
  • the 'edit cookie' panel will now not allow an ok if you accidentally paste a newline into any of these values
  • the 'edit header' panel now strips leading and trailing whitespace from the key and value
  • the 'edit header' panel will now not allow an ok if you accidentally paste a newline into either either

  • boring linting cleanup

  • now my IDE no longer has a cheeky multi-Qt env, its linter went nuts about old to-be-deprecated Enum references so I did more cleanup
  • moved from QDialog.Accepted/Rejected to the Qt6-only DialogCode Enum reference. there were about 400 of these I think
  • and `QFileDialog.AcceptMode`, `QFileDialog.FileMode`, and `QFileDialog.Option`
  • and `QLineEdit.EchoMode`
  • and `QAbstractItemView.SelectionMode`, `QAbstractItemView.SelectionBehavior`, and `QAbstractItemView.EditTrigger`
  • and `QSlider.TickPosition`
  • and `QFrame.Shadow` and `QFrame.Shape`
  • and `QSizePolicy.Policy`
  • and `QToolButton.ToolButtonPopupMode`
  • and `QTabWidget.TabPosition`
  • and I played around with typing.cast in a few places to handle some custom panels here and there. it is ok!
  • also figured out some nicer typing in my newer command-processing menu generation code, and filled in some places where the Command Processor Mixin was needed
  • also fixed some bad test panel stuff in the ancient lookup script panels

  • Qt when running from source

  • **I no longer support Qt5!** it may run, depending on version, if you set up your own venv, but my `setup_venv` scripts no longer offer it as a choice and I will no longer fix any new non-trivial Qt5 bugs. at some point I expect I will use a Qt6 technique for which there is no Qt5 equivalent and things will simply stop working
  • I cleaned up the Qt choice more in the `setup_venv` scripts, reducing it down to the one choice regarding Qt6 options and removing the '(m)iddle' choice in favour of simple old/new/test, and then a new '(q) for PyQt6' that just gets the latest PyQt6, and the '(w)rite your own'
  • the 'setup_venv' scripts now tell you that Python 3.13 is probably not going to work. they also say, in prep for when it will, in the '(w)rite your own' Qt version step, that Python 3.13's earliest version is 6.8.0.2. this is actually later than our current 'test' version, which is 6.7.something. I've now set up a 3.13 dev environment and did get the program booting but there seem to be problems with numpy<2.0.0. too, and then with scikit for psd-tools, which seems to have no Windows wheel, so I'll keep working here and update everything once I figure out something that will work out of the box. for now, assume python 3.13 is a no-go unless you know how to use pip. probably best to just wait six months for all the base stuff here to catch up and settle
  • I removed some Qt5 gubbins from the 'running from source' document

  • build stuff

  • updated a deprecated term in the Windows inno setup (the installer exe) user script
  • silenced a compiler warning about User-space-while-using-admin-installer in the Windows inno setup script. no good solution here, I think, but it isn't a huge deal

• version 600

  • misc

  • fixed a typoed type def that caused a boot error in python 3.10. this also sometimes/always broke the v599 macOS App, which I yanked. I have improved my testing situation to catch anything like this in future (ironically, I accidentally stopped booting the program in py 3.10 every Wednesday just recently when I stopped testing for Qt5) (issue #1630)
  • brought back the 'show more text about one file on the status bar' functionality with the new 'show additional status bar text when a single thumbnail is selected' checkbox under `options->thumbnails`. this tech works with single-file collections now, too. **this checkbox is default off, so go hit it if you miss this!**
  • the 'manage times' dialog now also shows '(3 days ago)' accompanying string variants with all the times displayed. time objects can get complicated, so let me know how this works out if you have weird times from the future or whatever
  • if your similar files search progress is greater than 99%, the tab no longer says (needs work). don't worry about that tiny little bit on every file import lads, the ride never ends
  • the 'open similar files in a new page' file menu now has a 'custom' entry that spawns a simple spinner to choose a different hamming distance. it starts at 10 and has step 2 (i.e. 10, 12, 14...). **related:** I understand there are sometimes small file count differences between a hamming distance of (even number) n and (odd) n+1, but it typically isn't significant. I force a hamming distance step of 2 in some places and allow a step of 1 in others. if you care about this stuff, would you be ok with me enforcing step 2 everywhere, or do you have use for odd-number hamming distance searches?
  • the various core file copy/move operations used in hydrus are now wrapped in retry modes for `BlockingIOError`, which can be raised by NASes and the like that are under deep stress. these will be retried five times, with ~1-3 second delays between re-attempts, before raising the Exception as before. thanks to the user who pointed out this could even happen. let me know if anything more complicated, let's say folder mirrors/merges, can still trip the problem
  • the 'analyze' database maintenance job's yes/no dialog is now a yes/yes/no dialog. the 'soft' and 'full' options are green, the cancel is red
  • cleaned up how hidden or non-functional galleries are chosen in the gallery-selector. the respective entries now have ellipses, and if there is only one item to select, it still shows the 'select from list' thing rather than confusingly insta-selecting that item
  • collections now sort by anything involving width and height by using its largest-num-pixels image as proxy. this isn't perfect, and it is invisible, and I suspect in particularly crazy situations the sort may change depending on the previous sort, but collections should at least do something here now rather than always counting as 'none' for their dimensions. a thought was floated to optionally sort collections by 'average' file property in places, which may be another avenue to explore here--for instance, although sum num_pixels might make some sense, sum of height doesn't
  • the 'sort by dimensions' submenu now lists width first, then height. a three-part hack was needed to make this work lol

  • more vacuum and db stuff

  • last week the vacuum stuff went well, but I forgot to update the dialog to use the new 'do we have enough free space to do a vacuum?' check! the dialog now properly uses the new check and won't moan about you not having enough temporary space
  • if you try to do a vacuum with an external program connected to the database file we want to vacuum (it tests this by looking for '-wal' or '-journal' journal sidecars), the program now abandons the vacuum attempt and tells you to disconnect and try again (or switch to WAL, if you are in TRUNCATE). one user did this by accident this week, with a pending write commit from the external program, and it caused malformation!! hydrus will now not let this happen
  • if the client or server have a failed vacuum due to a file rename failing and do not fix it themselves, and the program thus next boots with a db file with the `db_path.prevacuum` filename, the program now recognises this, tells you about it, automatically recovers from the situation, and, if one exists, tells you what to do with the `db_path.vacuum` file
  • the 'do we have a missing database file?' initialisation check is improved: it will now report on a mix of missing main and external files; it will now differentiate between the first connection attempt and subsequent normal reconnections (and only run 'create db' checks and so on on the first attempt); and it will recognise if a database file suddenly goes missing during program operation and trigger an immediate program halt after its popup
  • the various places the database can trigger an immediate program halt are now formalised into one careful method in a special place that won't be accidentally shuffled around or called via typo
  • I rejiggered the 'let's map our incoming db command to the bound method call' process and finally replaced the comp-sci-tier 80-line-tall if/else towers in the main `ClientDB` read/write methods. it is all now a much nicer dynamically initialised name->method dict. I don't suppose it really saves all that much overhead per call, but it is finally done
  • the database now reports 'db committing' in the main gui status bar when it does this in its 'idle' time. previously it only reported after a big job overran the due time

  • more hover windows

  • I may have fixed another hover window position bug (most probably certain window managers in Linux only, where sometimes windows will defer geometry updates until they are shown), where, before being shown any other time in that media viewer, it could flash into place in the correct position for one frame despite the mouse not being over it
  • simplified the hover window show/hide logic a bit more and removed an unusual hack that handled Window Managers that weren't happy about taskbarless top level hover windows appearing (because hover windows are no longer top level but just internal widgets to the normal media viewer)
  • re-integrated some spammy hover window reporting text into the 'hover window report mode', and rejiggered the logic to ensure the critical show/hide calls here are not so spammy

  • client api

  • added a new command, `/manage_database/force_commit`, which immediately and synchronously commits the database, flushing all pending (savepointed) changes to disk
  • added help for this: https://hydrusnetwork.github.io/hydrus/developer_api.html#manage_database_force_commit
  • the client api version is now 76

  • debugging

  • added a new `help->debug->report modes->file sort report mode`, which spams the thumbnail sorts going on with all the `sort_key` results it saw. we're trying to figure out some weird namespace sort, so let's see what this generates
  • file and search logs have a new advanced menu item, tucked under the 'whole log' submenu, to export the current selection to the clipboard in JSON serialised format

  • boring cleanup

  • fixed some more unresolved references caused by Qt enums

  • new mpv and sqlite on Windows

  • I am rolling in two new dlls for Windows today, for SQLite (database) and libmpv (video/audio player)
  • SQLite is updated from 3.45.3 to 3.47.0
  • mpv is updated from 2023-08-20 to 2024-10-20
  • both dlls are mostly just bugfixes and performance improvements, but the mpv release is slightly special--in the 'future test' we ran a few weeks ago, users with unusual Windows, be that Windows Server, under-updated Windows 10, or Windows 10 on a VM, might see a grid of black bars over some webms. no one on Windows 11 or normal updated Windows 10 reported any problems. the new mpv does perform much better than the older, and I am told it fixes some gif bugs, so I do want to update, but I do so hesitantly. if many users on updated/normal Windows do run into trouble with this release, I expect to roll back again. in the meantime, I have updated the 'running from source' help to talk more about stable versions of mpv on older Windows. users who need to keep their OS under-updated are now recommended to run from source: https://hydrusnetwork.github.io/hydrus/running_from_source.html

• version 599

  • misc

  • some users ended up getting a crazy wide duplicates page after the recent mix-up with the new auto-resolution multi-column list's ID. on update, the list will be reset to default widths, which should fix the rest of the duplicates page. let me know if you have any more trouble! (issue #1625)
  • the e621 downloader is fixed to find search results again (I updated the gallery parser to reflect their recent html changes). I understand they may be updating again soon, so let me know if anything breaks again (issue #1628)
  • I cleaned the media viewer hover window show/hide and size/position code a bit more and think I reduced more layout flicker, particularly the 'do another adjustment layout right after showing' issue the notes hover often does when super tall. I _might_ have also fixed the 'ok I guess I am going to sometimes initialise as a super wide guy in the middle of the viewer and when the user moves the mouse over my ghost I will flicker for one frame before moving where I belong' issue too
  • the timestamps in the media viewer top hover window and all the timestamp lines in the media right-click menu's top-row submenu now have a tooltip that has times with the reverse of your 'always show ISO times' settings. if you see '2022-11-20 14:39:52', it'll say '2 years ago' on the tooltip, and vice versa!
  • when you hit "open files in a new duplicates filter page", the file domain is now set explicitly to "all my files". if you know what you are doing and need to turn this off, there's a new checkbox under `options->duplicates`
  • new duplicates filter pages now also start on "all my files" rather than your default fallback file domain. "all my files" is the ideal default duplicates search context bros
  • if you have an OR predicate under construction in a file search tag autocomplete (use Shift+Enter when you enter stuff), hitting Escape on an empty text input now 'rewinds' it one predicate at a time before cancelling it entirely
  • animations that report they have exactly 100fps are now distrusted and their frames are counted manually. this 'count the frames manually' routine will now trigger generally in more cases and on larger files
  • all files with ~100fps will be metadata-rescanned on update

  • new vacuum tech

  • the vacuum in the client and server now uses `VACUUM INTO` SQLite tech. rather than writing a copy of the db to your temp dir and then to the WAL and then writing that 'commit' into the original file (which it seems is actually what happens--there's no atomic filename swap at the end), we now vacuum to a fresh file beside the original, no temp dir or WAL gubbins needed, and do a filename swap afterwards. this appears to work significantly faster than the old method, with the only caveat that there is a very brief dangerzone where neither file is named what we need, so if the hard drive disconnects or similar in that 2ms window, hydrus cannot automatically recover itself
  • all operations err on the side of failsafe, and I have added copious error handling code to navigate all the possible problems. if anything does go wrong, the user will be presented with a record of what happened, and in the case where hydrus could not fix itself, how they can fix it
  • the various 'do we have enough space to vacuum?' and 'how long will it take?' tests are adapted to the new rules. it doesn't use the temp dir any more, so if you have been struggling to find system drive space for client.mappings.db, this is no longer a problem
  • when the job is complete, the log message, which previously just said the time taken, now records the size change and the bytes per second, and this message is also now thrown into a popup. on my dev machine SSD, I have seen 170MB/s on database files that are in memory and ~30MB/s on a 4GB file not in memory. I guess we now asymptote to about 10MB/s on a superhuge file. I am interested in what users see in their different situations
  • If you are on an HDD and have a big db, you still have no hope, its over
  • thanks to the user who suggested this option

  • duplicates auto-resolution

  • _the duplicates auto-resolution system is moving forward. still lots to do, but I'm still feeling good about it all_
  • fleshed out the objects more--most stuff is now serialisable and has typedefs, getter/setters, and summary generation methods
  • connected most of the decision pipeline together, and we are basically ready to process our initial jpeg/png pixel pairs
  • did a hair more UI
  • my taglists now handle setEnabled calls properly, so the stub UI in the duplicates auto-resolution panel is now properly un-editable

  • boring cleanup

  • overhauled the way medias produce the various nice info strings on the right-click menu top-row submenu (and some other places). rather than a tangle of tuples, there's a couple simple classes being passed around that can do tooltip overrides and stuff. I also cleaned up the code around here generally
  • the main gui window status bar info when you only have one file selected is now much simpler. I previously piped the 'interesting' file info lines to it, but it too often ended up a spammy huge long line--not a good summary!
  • rewrote the 'always show iso time' solution from the old BaseMethod-to-Method replace trick to a simpler and saner global bool, as I recently did with some PIL/ICC settings
  • import folders now 'action' their original files immediately after import is done. previously they would do it in batches of ten, and if the import folder were interrupted by something like program shutdown, they'd have to wait for the next run to be cleared. I'm not totally sure, but I also think import folders set to 'ignore, not try again' on a large number of files may run a bit faster now
  • clarified the additional ways to import downloaders in the Lain import dialog. Lain's paste button now also accepts URI-aware file paths (i.e. if you select some files in your file explorer and tell your OS to 'copy') as the clipboard source
  • converted some duplicates processing code, and, relatedly, some delete-lock reporting stuff, from the being-overhauled `MediaSingleton` to `MediaResult`. this allowed me to clean up some wew code in duplicates auto-resolution, the duplicates filter, test code, and the Client API
  • updated the running from source help to talk about mpv/libmpv on Linux
  • fixed some bad panel/dialog calls into nicer Qt signals
  • added some typedefs to clear out about a hundred more PyUnresolvedReferences that PyCharm found. mostly custom widget calls

• version 598

  • misc

  • I screwed up the import folder deduplication fix last week! it caused import folders that contained duplicated items (and a handful of subscriptions, and even one normal GUI session) to not be able to save back their work. nothing was damaged, _per se_, but progress was not being saved and work was stopping after the respective systems paused out of safety. I am sorry for the trouble and worry here, and I hate it when this kind of error happens. I did made a test to test this thing worked, but it wasn't good enough. I have fixed it now and I am rejigging my test procedures to explicitly check for this specific class of object type problem (issue #1624)
  • fixed the duplicate filter comparison statements to obey the new 'do not use pretty (720p etc..) resolution swap-in strings' option (issue #1621)
  • the 'maintenance and processing' page now has some expand/collapse stuff on its boxes to make the options page not want to be so tall on startup
  • the 'edit filename tagging options' panel under the 'edit import folder' dialog now auto-populates the example filename from the actual folder's current contents. thanks to a user for pointing this out
  • moved a bunch of checkboxes around in the options. `options->tags` is renamed `tag autocomplete tabs` and now just handles children and favourites. `search` is renamed `file search` and handles the 'read' autocomplete and implicit system:limit, and a new page `tag editing` is added to handle the 'write' autocomplete and various 'tag service panel' settings
  • the normal search page 'selection tags' list now only computes the tags for the first n thumbnails (default 4096) on a page when you have no files selected. this saves time on mega pages when you click off a selection and also on giant import pages where new files are continually streaming in at the end. I expect this to reduce CPU and lag significantly on clients that idle looking at big import pages. you can set the n under `options->tag presentation`, including turning it off entirely. I did some misc optimisation here too, but I also found some places I can improve the general tag re-compute in future cleanup work
  • I may have improved some media viewer hover window positioning, sizing, and flicker in layout, particularly on the note window
  • the 'do really heavy sibling and parents calculation work in the background' daemon now waits 60 seconds after boot to start work (previously 10s). since I added the new fast sibling and parent cache (which works quick but takes some extra work to initialise), I've noticed you often get a heap of lag as this guy is initially populated right after boot. so, the primary caller now happens a little later in the boot rush and _should_ smooth out the curve a little

  • listbooks

  • I rewrote the 'ListBook' the options dialog relies on from ancient and irll-desingned wx code to a nice clean simple Qt panel
  • if you have a ton of tag services, a new 'use listbook instead of tabbed notebook for tag service panels' checkbox under `options->tag editing` now lets you use the new listbook instead of the old notebook/tabbed widget in: manage tags, manage tag siblings, manage tag parents, manage tag display and application, and review tag display sync

  • drag and drops

  • moved the DnD options out of `options->gui` and to a new `exporting` panel and added a bit of text
  • the BUGFIX 'secret' Discord fix is now formalised into an always-on 'set the DnD to a move flag', with a nice explanatory tooltip. it is now also always safe because it will now only ever run if you are set to export your DnDs to the temp folder
  • the 'DnD temp folder' system is now cleaner and DnD temp folders will now be deleted after six hours (previously they were only cleaned up on client exit)
  • added a note to the 'getting started with files' help to say you can export files with drag and drop m8

  • some multi-column list fixes

  • fixed a bad list type definition in the new auto-resolution rules UI. it thought it was the export folder dialog's list and was throwing weird errors if that list was sorted in column >=4
  • if a multi-column list fails to sort, it now catches and displays the error and continues with whatever was going on at the time
  • if a multi-column list status is asked for a non-existing column type, the status now reports the error info and attempts its best fallback
  • improved multi-column list initialisation across the board so the above problem cannot happen again (the list type was being set in two different locations, and I missed a ctrl+c/v edit)

  • parsing

  • behind the scenes, the 'subsidiary page parser' is now one object. it was a janky thing before
  • the subsidiary page parsers list in the parsing edit UI now has import/export/duplicate buttons
  • it doesn't matter outside of file log post order, I don't think, but subsidiary page parsers now always work in alphabetical order
  • they also now name themselves specifically when they cause an error
  • parsers now deduplicate the list when saying what they 'produce/parse' in UI

  • boring linting cleanup

  • tweaked my linter settings to better catch some stupid errors and put the effort into cleaning up the hundreds of long-time warnings, probably more than a thousand items of Qt Signal false-positive spam, and the actual real bugs. I am hoping to better expose future needles without a haystack of garbage in the way. I am determined to maintain a 0 error count on Unresolved References going forward
  • every single unused import statement is now removed or suppressed. I'm sure there are still tangles and bad ideas generally, but everything is completely lean now
  • fixed some PILImage enum references
  • improved some hydrus serialisable typedefs
  • fixed some exception/warning defs
  • deleted some old defunct 'retry' code from subscriptions
  • fixed some bitmap generation code to handle non-c-contiguous memoryviews properly
  • cleaned up some html parsing to properly navigate weird stuff bs4 might put out
  • fixed a stupid type error in the old HydrusTagArchive namespace code
  • fixed some account type calls in _manage services_ auto-account creation
  • fixed an issue with unusual tab drag and drops
  • deleted the empty `TestClientData.py`
  • deleted the empty `ServerServices.py`
  • fixed a bunch of misc typedefs in general

  • boring build/source stuff

  • updated my Windows 'running from source' help to now say you need to put the sqlite3.dll in your actual python DLLs dir. as this is more scary than just dropping it in your hydrus install dir, I emphasise this is optional
  • updated my 'requirements_server.txt', which is not used but a reference, to use the new requests and setuptools versions we recently updated to
  • I am dropping support for the ancient OpenCV 2. we've had some enum monkeypatches in place for years and years, but I don't even know if 2 will even run on any modern python; it is gone now

• version 597

  • misc

  • fixed an issue that caused non-empty hard drive file import file logs that were created before v595 (this typically affected import folders that are set to 'leave source alone, do not reattempt it' for any of the result actions) to lose track of their original import objects' unique IDs and thus, when given more items to possibly add (again, usually on an import folder sync), to re-add the same items one time over again and essentially double-up in size one time. this broke the ability to review the file log UI panel too, so users who noticed the behaviour was jank couldn't see what was going on. on update, all the newer duplicate items will be removed and you'll reset to the original 'already in db' etc.. stuff you had before. all file logs now check for and remove newer duplicates whenever they load or change contents. this happened because of the 'make file logs load faster' update in v595--it worked great for downloaders and subs, but local file imports use a slightly different ID system to differentiate separate objects and it was not updated correct
  • the main text-fetching routine that failed to load the list UI in the above case can now recover from null results if this happens again
  • file import objects now have some more safety code to ensure they are identifying themselves correctly on load
  • did some more work on copying tags: the new 'always copy parents with tags' was not as helpful as I expected, so this is no longer the default when you hit Ctrl+C (it goes back to the old behaviour of just copying the top-line rows in your selection). when you open a tag selection 'copy' menu, it now lists as a separate item 'copy 2 selected and 3 parents' kind of thing if you do want parents. also, parents will no longer copy with their indent (wew), and the taglists are now deduped so you will not be inundated with tagspam. futhermore, the 'what tags do we have' taglist in the manage tags dialog, and favourites/suggestions taglists, are now more parent-aware and plugged into this system
  • added Mr Bones to the frame locations list under `options->gui`. if you use him a lot, he'll now remember where he was and how big he was
  • also added `manage_times_dialog`, `manage_urls_dialog`, `manage_notes_dialog`, and `export_files_frame` to the list. they will all remember last size and position by default
  • the client now recovers from a missing frame location entry with a fallback and a note in the log
  • rewrote the way the media viewer hover windows and their sub-controls are updated to the current media object. the old asynchronous pubsub is out, and synchronous Qt signals are in. fingers crossed this truly fixes the rare-but-annoying 'oh the ratings in the top-right hover aren't updating I guess' bug, but we'll see. I had to be stricter about the pipeline here, and I was careful to ensure it would be failsafe, so if you discover a media viewer with hover windows that simply won't switch media (they'd probably be frozen in a null state from viewer open), let me know the details!
  • some built versions of the client seem unable to find their local help, so now, when a user asks to open a help page, if it seems to be missing locally, a little text with the paths involved is now written to the log

  • parsing

  • all formulae now have a 'name/description' field. this is wholly decorative and simply appears in the single- or multi-line summary of the formula in UI. all formulae start with and will initialise with a blank label
  • the generic 'edit formula' panel (the one where you can change the formula type) now has import/export buttons
  • updated the ZIPPER UI to use a newer single-class 'queue list' widget rather than some ten year old 'still has some wx in it' scatter of gubbins
  • added import/export/duplicate capability to the 'queue list' widget, and added it for ZIPPER formulae
  • also added import/export/duplicate buttons to the 'edit string processor' list!!
  • 'any characters' String Match objects now describe themselves with the 'such as' respective example string, with the new proviso that no String Match will give this string if it is stuck at the 'example string' default. you'll probably most see this in the manage url class dialog for components and parameters
  • cleaned a bunch of this code generally

  • client api

  • fixed an issue fetching millisecond-precise timestamps in the `file_metadata` call when one of the timestamps had a null value (for instance if the file has no modified date of any kind registered)
  • in the various potential duplicates calls, some simple searches (usually when one/both of two searches are system:everything) are now optimised using the same routine that happens in UI
  • the client api version is now 75

  • Win 7 news

  • for Win 7 users who run from source, I believe newer the program's newer virtual environments will no longer build in Win 7. it looks like a new version of psd-tools will not compile in python 3.8, and there's also some code in newer versions of the program that 3.8 simply won't run. I think the last version that works for you is v582. we've known this train was coming for a while, so I'm afraid Win 7 guys will have to freeze at that version unless and until they update Windows or move to Linux/macOS
  • I have updated the 'running from source' help to talk about this, including adding the magic git line you need to choose a specific version rather than normal git pull. this is likely the last time I will specifically support Win 7, and I suspect I will sunset pyside2 and PyQt5 testing too

  • Windows future build

  • I am releasing a future build alongside this release, just for Windows. it has new dlls for SQLite and mpv. advanced users are invited to test it out and tell me if there are any problems booting and playing media, and if there are no issues, I'll fold this into the normal build next week
  • mpv: 2023-08-20 to 2024-10-20
  • SQLite: 3.45.3 to 3.47.0
  • these bring normal optimisations and bug fixes. I expect no huge problems (although I believe the mpv dll strictly no longer supports Win 7, but that is now moot), but please check and we'll see

  • boring code cleanup

  • in prep for duplicates auto-resolution, the five variables that go into a potential duplicates search (two file searches, the search type, the pixel dupe requirement, and the max hamming distance) are now bundled into one nice clean object that is simpler to handle and will be easier to update in future. everything that touches this stuff--the page manager, the page UI (there's a whole edit panel for the new class), the filter itself, the Client API, the db search code, all the unit tests, and now the duplicates auto-resolution system--all works on this new thing rather than throwing list of variables around

  • duplicates auto-resolution

  • I pushed this forward in a bunch of ways. nothing actually works yet, still, but if you poke around in the advanced placeholder UI, you'll see the new potential duplicates search context UI, now with side-by-side file search context panels, for the fleshed-out pixel-perfect jpeg/png default

• version 596

  • misc

  • due to an ill-planned parsing update, several downloaders' hash lookups (which allow the client to quickly determine 'already in db'/'previously deleted' sometimes) broke last week. they are fixed today, sorry for the trouble!
  • the fps number on the file info line, which was previously rounded always to the nearest integer, is now reported to two sig figs when small. it'll say 1.2fps and 0.50fps
  • I hacked in some collapse/expand tech into my static box layout that I use all over the place and tentatively turned it on, and defaulting to collapsed, in the bigger _review services_ sub-panels. the giganto-tall repository panel is now much shorter by default, making the rest of the pages more normal sized on first open. let's see how it goes, and I expect I'll put it elsewhere too and add collapse memory and stuff if that makes sense
  • the 'copy service key' on _review services_ panels is now hidden behind advanced mode
  • tweaked some layout sizers for some spinboxes (the number controls that have an up/down arrow on the side) and my 'noneable' spinboxes so they aren't so width-hesitant. they were not showing their numbers fully on some styles where the arrows were particularly wide. they mostly size stupidly wide now, but at least that lines up with pretty much everything else so the number of stupid layout problems we are dealing with has reduced by one
  • the frame locations list under `options->gui` has four new buttons to mass-set 'remember size/position' and 'reset last size/position' to all selected
  • max implicit system:limit in `options->search` is raised from 100 thousand to 100 million
  • if there is a critical drive problem when adding a file to the file structure, the exact error is now spammed to a popup and log. previously, it was just propagated up to the caller

  • advanced parsing

  • I messed up the 'hex' and 'base64' decode stuff last week. we used to have hex and base64 decode back in python 2 to do some hash conversion stuff, but it was overhauled into the content parser hash type dropdown and the explict conversion was deprecated to a no-op. last week, I foolishly re-used the same ids when I revived the decoding functionality, which caused a bunch of old parsers like gelbooru 0.2.5, e621, 4chan, and likely others, which still had the no-op, to suddenly hex- or base-64-afy their parsed hashes, breaking the parse and lookup
  • this week I redefined the hacky enums and generally cleaned this code, and **I am deleting all hex and base64 string conversion decodes from all pre-596 parsers**. this fixes all the old downloaders by explicitly deleting the no-op so it won't trouble us again
  • if you made a string converter in v595 that decodes hex or base64, that encoding step will be deleted, sorry! I have to ask you to re-make it

  • advanced db maintenance

  • added a 'connect.bat' (and .sql file) to the db dir to make it easy to load up the whole database with 'correct' ATTACHED schema names in the sqlite3 terminal
  • added `database->db maintenance->get tables using definitions`, which uses the long-planned database module rewrite maintenance tech ( basically a faux foreign key) to fetch every table that uses hash_ids or tag_ids along with the specific column name that uses the id. this will help with various advanced maintenance jobs where we need to clear off a particular master definition to, as for instance happened this week, reset a super-huge autoincrement value on the master hashes table. this same feature will eventually trim client.master.db by discovering which master definitions are no longer used anywhere (e.g. after PTR delete)

  • client api

  • thanks to the continuing efforts of the user making Ugoira improvements, the Client API's `/get_files/render` call will now render an Ugoira to apng or animated webp. note the apng conversion appears to take a while, so make sure you try both formats to see what you prefer
  • fixed a critical bug in the Client API where if you used the `file_id(s)` request parameter, and gave novel ids, the database was hitting emergency repair code and filling in the ids with pseudorandom recovery hashes. this wasn't such a huge deal, but if you put a very high number in, the autoincrement `hash_id` of the hashes table would then move up to there, and if the number was sufficiently high, SQLite would have trouble because of max integer limits and all kinds of stuff blew up. asking about a non-existent `file_id` will now raise a 404, as originally intended
  • refactored the note set/delete calls, which were doing their own thing, to use the unified hash-parsing routine with the new safety code
  • if the Client API is ever asked about a hash_id that is negative or over a ~quadrillion (1024^5), it now throws a special error
  • as a backup, if the Client DB is ever asked about a novel hash_id that is negative or over a ~quadrillion (1024^5), it now throws a special error rather than trigger the pseudorandom hash recovery code
  • the Client API version is now 74

  • boring duplicates auto-resolution stuff

  • fleshed out the duplicates auto-resolution manager and plugged it into the main controller. the mainloop boots and exits now, but it doesn't do anything yet

  • boring cleanup

  • updated the multiple-file warning in the edit file urls dialog
  • gave the Client API _review services_ panel a very small user-friendliness pass
  • I converted more old multi-column list display/sort generation code from the old bridge to the newer, more efficient separated calls for 10 of the remaining 43 lists to do
  • via some beardy-but-I-think-it-is-ok typedefs, all the managers and stuff that take the controller as a param now use the new 'only import when linting' `ClientGlobals` Controller type, all unified through that one place, and in a way that should be failsafe, making for much better linting in any decent IDE. I didn't want to spam the 'only import when linting' blocks everywhere, so this was the compromise
  • deleted the `interface` modules with the Controller interface gubbins. this was an _ok_ start of an idea, but the new Globals import trick makes it redundant
  • pulled and unified a bunch of the common `ManagerWithMainLoop` code up to the superclass and cleaned up all the different managers a bit
  • deleted `ClientMaintenance.py`, which was an old attempt to unify some global maintenance daemons that never got off the ground and I had honestly forgotten about
  • moved responsibility for the `remote_thumbnails` table to the Client Repositories DB module; it is also now plugged into the newer content type maintenance system
  • moved responsibility for the `service_info` table to the Client Services DB module
  • the only CREATE TABLE stuff still in the old Client DB creation method is the version table and the old YAML options structure, so we are essentially all moved to the new modules now
  • fixed some bugs/holes in the table definition reporting system after playing with the new table export tool (some bad sibling/parent tables, wrongly reported deferred tables, missing notes_map and url_map due to a bad content type def, and the primary master definition tables, which I decided to include). I'm sure there are some more out there, but we are moving forward on a long-term job here and it seems to work

• version 595

  • ugoiras

  • thanks to a user who put in a lot of work, we finally have Ugoira rendering! all ugoiras will now animate using the hydrus native animation player. if the ugoira has json timing data in its zip (those downloaded with PixivUtil and gallery-dl will!), we will use that, but if it is just a zip of images (which is most older ugoiras you'll see in the wild), it'll check a couple of note names for the timing data, and, failing that, will assign a default 125ms per frame fallback. ugoiras without internal timing data will currently get no 'duration' metadata property, but right-clicking on them will show their note-based or simulated duration on the file info line
  • all existing ugoiras will be metadata rescanned and thumbnail regenned on update
  • technical info here: https://hydrusnetwork.github.io/hydrus/filetypes.html#ugoira
  • ugoira metadata and thumbnail generation is cleaner
  • a bug in ugoira thumbnail selection, when the file contains non-image files, is fixed
  • a future step will be to write a special hook into the hydrus downloader engine to recognise ugoiras (typically on Pixiv) and splice the timing data into the zip on download, at which point we'll finally be able to turn on Ugoira downloading on Pixiv on our end. for now, please check out PixivUtil or gallery-dl to get rich Ugoiras
  • I'd like to bake the simulated or note-based durations into the database somehow, as I don't like the underlying media object thinking these things have no duration, but it'll need more thought

  • misc

  • all multi-column lists now sort string columns in a caseless manner. a subscription called 'Tents' will now slot between 'sandwiches' and 'umbrellas'
  • in 'favourite searches', the 'folder' name now has hacky nested folder support. just put '/' in the folder name and it'll make nested submenus. in future this will be implemented with a nicer tree widget
  • file logs now load faster in a couple of ways, which should speed up UI session and subscriptions dialog load. previously, there were two rounds of URL normalisation on URL file import object load, one wasteful and one fixable with a cache; these are now dealt with. thanks to the users who sent in profiles of the subscriptions dialog opening; let me know how things seem now (hopefully this fixes/relieves #1612)
  • added 'Swap in common resolution labels' to `options->media viewer`. this lets you turn off the '1080p' and '4k'-style label swap-ins for common resolutions on file descriptor strings
  • the 'are you sure you want to exit the client? 3 pages say "I am still importing"' popup now says the page names, and in a pretty way, and it shows multiple messages nicer
  • the primary 'sort these tags in a human way m8' routine now uses unicode tech to sort things like ß better
  • the String Converter can decode 'hex' and 'base64' again (so you can now do '68656c6c6f20776f726c64' or 'aGVsbG8gd29ybGQ=' to 'hello world'). these functions were a holdover from hash parsing in the python 2 times, but I've brushed them off and cleared out the 'what if we put raw bytes in the parsing system bro' nonsense we used to have to deal with. these types are now explictly UTF-8. I also added a couple unit tests for them
  • fixed an options initialisation bug where setting two files in the duplicate filter as 'not related' was updating the A file to have the B file's file modified time if that was earlier!! if you have files in this category, you will be asked on update if you want to reset their file modified date back to what is actually on disk (the duplicate merge would not have overwritten this; this only happens if you edit the time in the times dialog by hand). a unit test now checks this situation. sorry for the trouble, and thank you to the user who noticed and reported this
  • the hydrus Docker package now sets the 'hydrus' process to `autorestart=unexpected`. I understand this makes `file->exit` stick without an automatic restart. it seems like commanding the whole Docker image to shut down still causes a near-instant unclean exit (some SIGTERM thing isn't being caught right, I think), but `file->exit` should now be doable beforehand. we will keep working here

  • more OR preds

  • the new 'replace selected with their OR' and the original 'add an OR of the selected' are now mutually exclusive, depending on whether the current selection is entirely in the active search list
  • added 'start an OR with selected', which opens the 'edit OR predicate' panel on the current selection. this works if you only select one item, too
  • added 'dissolve selected into single predicates', when you select only OR predicates. it does the opposite of the 'replace'
  • the new OR menu gubbins is now in its own separated menu section on the tag right-click
  • the indent for OR sub preds is moved up from two spaces to four

  • urls

  • wrote some help about the 'force page refetch' checkboxes in 'tag import options' here: https://hydrusnetwork.github.io/hydrus/getting_started_downloading.html#force_page_fetch
  • added a new submenu `urls->force metadata refetch` that lets you quickly and automatically create a new urls downloader page with the selected files' 'x URL Class' urls with the tag import options set to the respective URLs' default but with these checkboxes all set for you. we finally have a simple answer to 'I messed up my tag parse, I need to redownload these files to get the tags'!
  • the urls menu offers the 'for x url class' even when only one file is selected now. crazy files with fifty of the same url class can now be handled

  • duplicates auto-resolution

  • wrote some placeholder UI for the new system. anyone who happens to be in advanced mode will see another tab on duplicate filter pages. you can poke around if you like, but it is mostly just blank lists that aren't plugged into anything
  • wrote some placeholder help too. same deal, just a placeholder that you have to look for to find that I'll keep working on
  • I still feel good about the duplicates auto-resolution system. there is much more work to do, but I'll keep iterating and fleshing things out

  • client api

  • the new `/get_files/file_path` command now returns the `filetype` and `size` of the file
  • updated the Client API help and unit tests for this
  • client api version is now 73

  • new build stuff

  • the library updates we've been testing the past few weeks have gone well, so I am rolling them into the normal builds for everyone. the libraries that do 'fetch stuff from the internet' and 'help python manage its packages' are being updated because of some security problems that I don't think matter for us at all (there's some persistent https verification thing in requests that I know we don't care about, and a malicious URL exploit in setuptools that only matters if you are using it to download packages, which, as I understand, we don't), but we are going to be good and update anyway
  • `requests` is updated from `2.31.0` to `2.32.3`
  • `setuptools` is updated from `69.1.1` to `70.3.0`
  • `PyInstaller` is updated from `6.2` to `6.7` for Windows and Linux to handle the new `setuptools`
  • there do not appear to be any update conflicts with dlls or anything, so just update like you normally do. I don't think the new pyinstaller will have problems with older/weirder Windows, but let me know if you run into anything
  • users who run from source may like to reinstall their venvs after pulling to get the new libraries too

  • boring cleanup

  • refactored `ClientGUIDuplicates` to a new `duplicates` gui module and renamed it to `ClientGUIDuplicateActions`
  • harmonised some duplicates auto-resolution terminology across the client to exactly that form. not auto-duplicates or duplicate auto resolution, but 'duplicates auto-resolution'
  • fixed some bad help link anchors
  • clarified a couple things in the 'help my db is broke.txt' document
  • updated the new x.svg to a black version; it looks a bit better in light & dark styles

• version 594

  • misc

  • fixed an error that was stopping files from being removed sometimes (it also messed up thumbnail selection). it could even cause crashes! the stupid logical problem was in my new list code; it was causing the thumbnail grid backing list to get pseudorandomly poisoned with bad indices when a previous remove event removed the last item in the list
  • the tag `right-click->search` menu, on a multiple selection of non-OR predicates that exists in its entirely in the current search context, now has `replace selected with their OR`, which removes the selection and replaces it with an OR of them all!
  • the system predicate parser no longer removes all underscores from to-be-parsed text. this fixes parsing for namespaces, URLs, service names, etc.. with underscores in (issue #1610)
  • fixed some bad layout in the edit predicates dialog for system:hash (issue #1590)
  • fixed some content update logic for the advanced delete choices of 'delete from all local file domains' and 'physically delete now', where the UI-side thumbnail logic was not removing the file from the 'all my files' or 'all local files' domains respectively, which caused some funny thumbnail display and hide/show rules until a restart rebuilt the media object from the (correct) db source
  • if you physically delete a file, I no longer force-remove it from view so enthusiastically. if you are looking at 'all known files', it should generally still display after the delete (and now it will properly recognise it is now non-local)
  • I may have fixed an issue with page tab bar clicks on the very new Qt 6.8, which has been rolling out this week
  • wrote out my two rules for tagging (don't be perfect, only tag what you search) to the 'getting started - more tags' help page: https://hydrusnetwork.github.io/hydrus/getting_started_more_tags.html#tags_are_for_searching_not_describing

  • shutdown improvements

  • I cleaned up and think I fixed some SIGTERM and related 'woah, we have to shut down right now' shutdown handling. if a non-UI thread calls for the program to exit, the main 'save data now' calls are now all done by or blocked on that thread, with improved thread safety for when it does tell Qt to hide and save the UI and so on (issue #1601, but not sure I totally fixed it)
  • added some SIGTERM test calls to `help->debug->tests` so we can explore this more in future
  • on the client, the managers for db maintenance, quick downloads, file maintanence, and import folders now shut down more gracefully, with overall program shutdown waiting for them to exit their loops and reporting what it is still waiting on in the exit splash (like it already does for subscriptions and tag display). as a side thing, these managers also start faster on program boot if you nudge their systems to do something

  • boring cleanup

  • wrote some unit tests to test my unique list and better catch stupid errors like I made last week
  • added default values for the 'select from list of things' dialogs for: edit duplicate merge rating action; edit duplicate merge tag action; and edit url/parser link
  • moved `FastIndexUniqueList` from `HydrusData` to `HydrusLists`
  • fixed an error in the main import object if it parses (and desires to skip associating) a domain-modified 'post time' that's in the first week of 1970
  • reworked the text for the 'focus the text input when you change pages' checkbox under `options->gui pages` and added a tooltip
  • reworded and changed tone of the boot error message on missing database tables if the tables are all caches and completely recoverable
  • updated the twitter link and icon in `help->links` to X

• version 593

  • misc

  • in a normal search page tag autocomplete input, search results will recognise exact-text-matches of their worse siblings for 'put at the top of the list' purposes. so, if you type 'lotr', and it was siblinged to 'series:lord of the rings', then 'series:lord of the rings' is now promoted to the top of the list, regardless of count, as if you had typed in that full ideal tag
  • OR predicates are now multi-line. the top line is OR:, and then each sub-tag is now listed indented below. if you construct an OR pred using shift+enter in the tag autocomplete, this new OR does start to eat up some space, but if you are making crazy 17-part OR preds, maybe you'll want to use the OR button dialog input anyway
  • when you right-click an OR predicate, the 'copy' menu now recognises this as '3 selected tags' etc.. and will copy all the involved tags and handle subtags correctly
  • the 'remove/reset for all selected' file relationship menu is no longer hidden behind advanced mode. it being buried five layers deep is enough
  • to save a button press, the manage tag siblings dialog now has a paste button for the right-side tag autocomplete input. if you paste multiple lines of content, it just takes the first
  • updated the file maintenance job descriptions for the 'try to redownload' jobs to talk about how to deal with URL downloads that 404 or produce a duplicate and brushed up a bit of that language in general
  • the new 'if a db job took more than 15 seconds, log it' thing now tests if the program was non-idle at the start or end of the db job, rather than just the end. this will catch some 'it took so long that some "wake up" stuff had time to kick in' instances
  • fixed a typo where if the 'other' hashes were unknown, the 'sha512 (unknown)' label was saying 'md5 (unknown)'
  • file import logs get a new 'advanced' menu option, tucked away a little, to 'renormalise' their contents. this is a maintenance job to clear out duplicate chaff on an existing list after the respective URL Class rules have changed to remove something in normalisation (e.g. setting a parameter to be ephemeral). I added a unit test for this also, but let me know how it works in the wild

  • default downloaders

  • fixed the source time parsing for the gelbooru 0.2.0 (rule34.xxx and others) and gelbooru 0.2.5 (gelbooru proper) page parsers

  • client api

  • fixed the 'permits everything' API Permissions update from a couple weeks ago. it was supposed to set 'permits everything' when the existing permissions structure was 'mostly full', but the logic was bad and it was setting it when the permissions were sparse. if you were hit by this and did not un-set the 'permits everything' yourself in _review services_, you will get a yes/no prompt on update asking if you want to re-run the fixed update. if the update only missed out setting "permits everything" where it should have, you'll just get a popup saying it did them. sorry for missing this, my too-brief dev machine test happened to be exactly on the case of a coin flip landing three times on its edge--I've improved my API permission tests for future

  • duplicate auto-resolution progress

  • I got started on the db module that will handle duplicates auto-resolution. this started out feeling daunting, and I wasn't totally sure how I'd do some things, but I gave it a couple iterations and managed to figure out a simple design I am very happy with. I think it is about 25-33% complete (while object design is ~50-75% and UI is 0%), so there is a decent bit to go here, but the way is coming into focus

  • boring code cleanup

  • updated my `SortedList`, which does some fast index lookup stuff, to handle more situations, optimised some remove actions, made it more compatible as a list drop-in replacement, moved it to `HydrusData`, and renamed it to `FastIndexUniqueList`
  • the autocomplete results system uses the new `FastIndexUniqueList` a bit for some cached matches and results reordering stuff
  • expanded my `TemporerIntegerTable` system, which I use to do some beardy 'executemany' SELECT statements, to support an arbitrary number of integer columns. the duplicate auto-resolution system is going to be doing mass potential pair set intersections, and this makes it simple
  • thanks to a user, the core `Globals` files get some linter magic that lets an IDE do good type checking on the core controller classes without running into circular import issues. this reduced project-wide PyCharm linter warnings from like 4,500 to 2,200 wew
  • I pulled the `ServerController` and `TestController` gubbins out of `HydrusGlobals` into their own 'Globals' files in their respective modules to ensure other module-crawlers (e.g. perhaps PyInstaller) do not get confused about what they are importing here, and to generally clean this up a bit
  • improved a daemon unit test that would sometimes fail because it was not waiting long enough for the daemon to finish. I cut some other fat and it is now four or five seconds faster too

• version 592

  • misc

  • the 'read' autocomplete dropdown has a new one-click 'clear search' button, just beside the favourites 'star' menu button. the 'empty page' favourite is removed from new users' defaults
  • in an alteration to the recent Autocomplete key processing, Ctrl+c/Ctrl+Insert _will_ now propagate to the results list if you currently have none of the text input selected (i.e. if it would have been a no-op on the text input, we assume you wanted whatever is selected in the list)
  • in the normal thumbnail/viewer menu and _review services_, the 'files' entry is renamed to 'locations'. this continues work in the left hand button of the autocomplete dropdown where you set the 'location', which can be all sorts of complicated things these days, rather than just 'file service key selector'. I don't think I'll rename 'my files' or anything, but I will try to emphasise this 'locations' idea more when I am talking about local file domains etc.. in other places going forward; what I often think of as 'oh yeah the files bit' isn't actually referring to the files themselves, but where they are located, so let's be precise
  • last week's tag pair filtering in _tags->migrate tags_ now has 'if either the left or right of the pair have count', and when you hit 'Go' with any of the new count filter checkboxes hit, the preview summary on the yes/no confirmation dialog talks about it
  • any time a watcher subject is parsed, if the text contains non-decoded html entities (like '&gt;'), they are now auto-converted to normal chars. these strings are often ripped from odd places and are only used for user display, so this just makes that simpler
  • if you are set to remove trashed files from view, this now works when the files are in multpile local file domains, and you choose 'delete from all local file services', and you are looking at 'all my files' or a subset of your local file domains
  • we now log any time (when the client is non-idle) that a database job's work inside the transaction wrapper takes more than 15 seconds to complete
  • fixed an issue caused by the sibling or parents system doing some regen work at an unlucky time

  • default downloaders

  • thanks to user help, the derpibooru post parser now additionally grabs the raw markdown of a description as a second note. this catches links and images better than the html string parse. if you strictly only want one of these notes, please feel free to dive into _network->downloaders->defailt import options_ for your derpi downloader and try to navigate the 'note import options' hell I designed and let me know how it could be more user friendly

  • parsing system

  • added a new NESTED formula type. this guy holds two formulae of any type internally, parsing the document with the first and passing those results on to the second. it is designed to solve the problem of 'how do I parse this JSON tucked inside HTML' and _vice versa_. various encoding stuff all seems to be handled, no extra work needed
  • added Nested formula stuff to the 'how to make a downloader' help
  • made all the screenshot in the parsing formula help clickable
  • renamed the COMPOUND formula to ZIPPER formula
  • all the 'String Processor' buttons across the program now have copy and paste buttons, so it is now easy to duplicate some rules you set up
  • in the parsing system, sidecar importer, and clipboard watcher, all strings are now cleansed of errant 'surrogate' characters caused by the source incorrectly providing utf-16 garbage in a utf-8 stream. fingers crossed, the cleansing here will actually _fix_ problem characters by converting them to utf-8, but we'll see
  • thanks to a user, the JSON parsing system has a new 'de-minify json' parsing rule, which decompresses a particular sort of minified JSON that expresses multiply-referenced values using list positions. as it happened that I added NESTED formulae this week, I wonder if we will migrate this capability to the string processing system, but let's give it time to breathe

  • client api

  • fixed the permission check on the new 'get file/thumbnail local path' commands--due to me copy/pasting stupidly, they were still just checking 'search files' perm
  • added `/get_files/local_file_storage_locations`, which spits out the stuff in _database->move media files_ and lets you do local file access _en masse_
  • added help and a unit test for this new command
  • the client api version is now 72

  • some security/library updates

  • the 'old' OpenCV version in the `(a)dvanced` setup, which pointed to version 4.5.3.56, which had the webp vulnerability, is no longer an option. I believe this means that the program will no longer run on python 3.7. I understad Win 7 can run python 3.8 at the latest, so we are nearing the end of the line on that front
  • the old/new Pillow choice in `(a)dvanced` setup, which offered support for python 3.7, is removed
  • I have added a new question to the `(a)dvanced` venv setup to handle misc 'future' tests better, and I added a new future test for two security patches for `setuptools` and `requests`:
  • A) `setuptools` is updated to 70.3.0 (from 69.1.1) to resolve a security issue related to downloading packages from bad places (don't think this would ever affect us, but we'll be good)
  • B) `requests` is updated to 2.32.3 (from 2.31.0) to resolve a security issue with verify=False (the specific problem doesn't matter for us, but we'll be good)
  • if you run from source and want to help me test, you might like to rebuild your venv this week and choose the new future choice. these version increments do not appear to be a big deal, so assuming no problems I will roll these new libraries into a 'future' test build next week, and then into the normal builds a week after

  • boring code cleanup

  • did a bunch more `super()` refactoring. I think all `__init__` is now converted across the program, and I cleared all the normal calls in the canvas and media results panel code too
  • refactored `ClientGUIResults` into four files for the core class, the loading, the thumbnails, and some menu gubbins. also unified the mish-mash of `Results` and `MediaPanel` nomenclature to `MediaResultsPanel`

• version 591

  • misc

  • fixed a stupid oversight with last week's "move page focus left/right after closing tab" thing where it was firing even when the page closed was not the current tab!! it now correctly only moves your focus if you close the _current_ tab, not if you just middle click some other one
  • fixed the _share->export files_ menu command not showing if you right-clicked on just one file
  • cleaned some of the broader thumbnail menu code, separating the 'stuff to show if we have a focus' and 'stuff to show if we have a selection'; the various 'manage' commands now generally show even if there is no current 'focus' in the preview (which happens if you select with ctrl+click or ctrl+a and then right-click in whitespace)
  • the 'migrate tags' dialog now allows you to filter the sibling or parent pairs by whether the child/worse or parent/ideal tag has actual mapping counts on an arbitrary tag service. some new unit tests ensure this capability
  • fixed an error in the duplicate metadata merge system where if files were exchanging known URLs, and one of those URLs was not actually an URL (e.g. it was garbage data, or human-entered 'location' info), a secondary system that tried to merge correlated domain-based timestamps was throwing an exception
  • to reduce comma-confusion, the template for 'show num files and import status' on page names is now "name - (num_files - import_status)"
  • the option that governs whether page names have the file count after them (under _options->gui pages_) has a new choice--'show for all pages, but only if greater than zero'--which is now the default for new users

  • some boring code cleanup

  • broke up the over-coupled 'migrate tags' unit tests into separate content types and the new count-filtering stuff
  • cleaned up the 'share' menu construction code--it was messy after some recent rewrites
  • added some better error handling around some of the file/thumbnail path fetching/regen routines

  • client api

  • the client api gets a new permissions state this week: the permissions structure you edit for an access key can now be (and, as a convenient default, starts as) a simple 'permits everything' state. if the permissions are set to 'permit everything', then this overrules all the specific rules and tag search filter gubbins. nice and simple, and a permissions set this way will automatically inherit new permissions in the future. any api access keys that have all the permissions up to 'edit ratings' will be auto-updated to 'permits everything' and you will get an update saying this happened--check your permissions in _review services_ if you need finer control
  • added a new permission, `13`, for 'see local paths'
  • added `/get_files/file_path`, which fetches the local path of a file. it needs the new permission
  • added `/get_files/thumbnail_path`, which fetches the local path of a thumbnail and optionally the filetype of the actual thumb (jpeg or png). it needs the new permission
  • the `/request_new_permissions` command now accepts a `permits_everything` bool as a selective alternate to the `basic_permissions` list
  • the `/verify_access_key` command now responds with the name of the access key and the new `permits_everything` value
  • the API help is updated for the above
  • new unit tests test all the above
  • the Client API version is now 71

  • client api refactoring

  • the main `ClientLocalServerResources` file has been getting too huge (5,000 lines), so I've moved it and `ClientLocalServer` to their own `api` module and broken the Resources file up into core functions, the superclass, and the main verbs
  • fixed permissions check for `/manage_popups/update_popup`, which was checking for pages permission rather than popup permission
  • did a general linting pass of these easier-to-handle files; cleaned up some silly stuff

• version 590

  • misc

  • the 'check now' button in manage subscriptions is generally more intelligent and now offers questions around paused status: if _all_ the selected queries are DEAD, it now asks you if you want to resurrect them with a yes/no variant of the DEAD/ALIVE question (previously it just did it); if you are in _edit subscriptions_ and any of the selected subs are paused, it now asks you if you want to include them (and unpause) in the check now, and if not it reduces the queries examined for the DEAD/ALIVE question appropriately (previously it just did their queries, and did not unpause); in either _edit subscriptions_ or _edit subscription_, if any queries in the selection after any 'paused subs' or 'DEAD/ALIVE' filtering are paused, it asks you if you want to include (and unpause) them in the check now (previously it just did and unpaused them all)
  • if you shrink the search page's preview window down to 0 size (which it will suddenly snap to, and which is a silghtly different hide state to the one caused by double-left-clicking the splitter sash), the preview canvas will now recognise it is hidden and no longer load media as you click on thumbs. previously this thing was loading noisy videos in the background etc..
  • the `StringMatch` 'character set' match type now has 'hexadecimal characters' (`^[\da-fA-F]+$`) and 'base-64 characters' (`^[a-zA-Z\d+/]+={0,2}$`) in its dropdown choice
  • the 'gui pages' options panel now has 'when closing tabs, move focus (left/right)', so if you'd rather move left when middle-clicking tabs etc.., you can now set it, and if your style's default behaviour is whack and never moved to the right before despite you wanting it, now you can force it; it is now explicit either way. let me know if any crazy edge-case focus logic happens in this mode with nested page of pages or whatever
  • when you right-click a file, in the _share->copy hash_ menu, the md5, sha1, and sha512 hashes are now loaded from the database, usually in the milliseconds after the menu is opened, and shown in the menu labels for quick human reference. if your client does not have these hashes for the file, it says so
  • the 'share' thumbnail menu is now visible on non-local files. it is severely truncated, basically just shows copy hash/file_id stuff
  • wrote a 'Current Deleted Pending Petitioned' section for the Developer API to discuss how the states in the content storage system overlap and change in relation to various commands in the content update pipeline https://hydrusnetwork.github.io/hydrus/developer_api.html#CDPP It may be of interest to non-API-devs who are generally interested in what exactly the 'pending' state etc.. is
  • if the file import options in a hard drive import page currently imports to an empty location context (e.g. you deleted the local file service it wanted to import to), the import page now pauses and presents an appropriate error text. the URL importers already did this, so this is the hdd import joining them
  • this 'check we are good to do file work' test in the importer pages now in all cases pursues a 'default' file import options to the actual real one that will be used, so if your importer file import options are borked, this is now detected too and the importer will pause rather than fail everything in its file log
  • thanks to a user, fixed a typo bug in the new multi-column list work that was causing problems when looking at gallery logs that included mis-linked log entries. in general, the main 'turn this integer into something human' function will now handle errors better

  • default downloaders

  • _advanced/technical, tl;dr: x.com URLs save better now._ since a better fix will take more work, the 'x post' URL class is for now set to associate URLs. this fixes the association of x.com URLs when those are explicitly referred to as source URLs in a booru post. previously, some hydrus network engine magic related to how x URLs are converted to twitter URLs (and then fx/vxtwitter URLs) to get parsed by the twitter parser was causing some problems. a full 'render this URL as this URL' system will roll out in future to better handle this situation where two non-API URLs can mean the same thing. this will result in some twitter/x post URL duplication--we'll figure out a nice merge later!

  • duplicate auto-resolution tech

  • I have written the first skeleton of the `MetadataConditional` object. it has a rule based on a system predicate (like 'width > 400px') and returns True/False when you give it a media object. this lego-brick will plug into a variety of different systems in future, including the duplicate auto-resolution system, with a unified UI
  • system predicates cannot yet do this arbitrarily, so it will be future work to fill out this code. to start with, I've just got system:filetype working to ultimately effect the first duplicate auto-resolution job of 'if pixel duplicates and one is jpeg, one png, then keep the jpeg'
  • add some unit tests to test this capability

  • boring search object code decoupling

  • refactored the main `Predicate` object and friends to the new `ClientSearchPredicate`
  • refactored the main `NumberTest` object and friends to the new `ClientNumberTest`
  • refactored the main `TagContext` object and friends to the new `ClientTagContext`
  • refactored the main `FileSearchContext` object and friends to the new `ClientSearchFileSearchContext`
  • moved some other `ClientSearch` stuff to other places and renamed the original file to `ClientSearchFavourites`; it now just contains the favourite searches manager
  • some misc cleanup around here. some enums had bad names, that sort of thing

• version 589

  • misc

  • the similar-files search maintenance code has an important update that corrects tree rebalancing for a variety of clients that initialised with an unlucky first import file. in the database update, I will check if you were affected here and immediately optimise your tree if so. it might take a couple minutes if you have millions of files
  • tag parent and sibling changes now calculate faster at the database level. a cache that maintains the structure of which pairs need to be synced is now adjusted with every parent/sibling content change, rather than regenerated. for the PTR, I believe this will save about a second of deferred CPU time on an arbitrary parent/sibling change for the price of about 2MB of memory, hooray. fingers crossed, looking at the _tags->sibling/parent sync->review_ panel while repository processing is going on will now be a smooth-updating affair, rather than repeated 'refreshing...' wait-flicker
  • the 'the pairs you mean to add seem to connect to some loops' auto-loop-resolution popup in the manage siblings/parents dialogs will now only show when it is relevent to pairs to be added. previously, this thing was spamming during the pre-check of the process of the user actually breaking up loops by removing pairs
  • added an item, 'sync now', to the _tags->sibling/parent sync_ menu. this is a nice easy way to force 'work hard' on all services that need work. it tells you if there was no work to do
  • reworked the 'new page chooser' mini-dialog and better fixed-in-place the intended static 3x3 button layout
  • showing 'all my files' and 'local files' in the 'new page chooser' mini-dialog is now configurable in _options->pages_. previously 'local files' was hidden behind advanced mode. 'all my files' will only ever show if you have more than one local files domain
  • when a login script fails with 401 or 403, or indeed any other network error, it now presents a simpler error in UI (previously it could spam the raw html of the response up to UI)
  • generally speaking, the network job status widget will now only show the first line of any status text it is given. if some crazy html document or other long error ends up spammed to this thing, it should now show a better summary
  • the 'filename' and 'first/second/etc.. directory' checkbox-and-text-input controls in the filename tagging panel now auto-check when you type something in
  • the 'review sibling/parent sync' and 'manage where tag siblings and parents apply' dialogs are now plugged into the 'default tag service' system. they open to this tab, and if you are set to update it to the last seen, they save over the value on changes

  • default downloaders

  • fixed the default safebooru file page parser to stop reading undesired '?' tags for every namespace (they changed their html recently I think)
  • catbox 'collection' pages are now parseable by default

  • boring list stuff

  • fixed an issue with showing the 'manage export folders' dialog. sorry for the trouble--in my list rewrite, I didn't account for one thing that is special for this list and it somehow slipped through testing. as a side benefit, we are better prepped for a future update that will support column hiding and rearranging
  • optimised about half of the new multi-column lists, as discussed last week. particularly included are file log, gallery log, watcher page, gallery page, and filename tagging panel, which all see a bunch of regular display/sort updates. the calls to get display data or sort data for a row are now separate, so if the display code is CPU expensive, it won't slow a sort
  • in a couple places, url type column is now sorted by actual text, i.e. file url-gallery url-post url-watchable url, rather than the previous conveniently ordered enum. not sure if this is going to be annoying, so we'll see
  • the filename tagging list no longer sorts the tag column by tag contents, instead it just does '#''. this makes this list sort superfast, so let's see if it is super annoying, but since this guy often has 10,000+ items, we may prefer the fast sort/updates for now

  • client api

  • the `/add_files/add_file` command now has a `delete_after_successful_import` parameter, default false, that does the same as the manual file import's similar checkbox. it only works on commands with a `path` parameter, obviously
  • updated client api help and unit tests to test this
  • client api version is now 70

  • more boring cleanup

  • I cleaned up a mash of ancient shortcut-processing jank in the tag autocomplete input and fixed some logic. everything is now processed through one event filter, the result flags are no longer topsy-turvy, and the question of which key events are passed from the text input to the results list is now a simple strict whitelist--basically now only up/down/page up/page down/home/end/enter (sometimes)/escape (sometimes) and ctrl+p/n (for legacy reasons) are passed to the results list. this fixes some unhelpful behaviour where you couldn't select text and ctrl+c it _unless_ the results list was empty (since the list was jumping in, after recent updates, and saying 'hey, I can do ctrl+c, bro' and copying the currently selected results)
  • the key event processing in multi-column lists is also cleaned up from the old wx bridge to native Qt handling
  • and some crazy delete handling in the manage urls dialog is cleaned up too
  • the old `EVT_KEY_DOWN` wx bridge is finally cleared out of the program. I also cleared out some other old wx event definitions that have long been replaced. mostly we just have some mouse handling and window state changes to deal with now
  • replaced many of my ancient static inheritance references with python's `super()` gubbins. I disentangled all the program's multiple inheritance into super() and did I think about half of the rest. still like 360 `__init__` lines to do in future
  • a couple of the 'noneable text' widgets that I recently set to have a default text, in the subscription dialogs, now use that text as placeholder rather than actual default. having 'my subscription' or whatever is ok as a guide, but when the user actually wants to edit, having it be real text is IRL a pain
  • refactored the repair file locations dialog and manage options dialog and new page picker mini-dialog to their own python files

• version 588

  • fast new lists

  • tl;dr: big lists faster now. you do not need to do anything
  • every multi-column list in the program (there's about 75 of them) now works on a more sophisticated model (specifically, we are updating from QTreeWidget to QTreeView). instead of the list storing and regenerating display labels for every single row of a table, only the rows that are currently in view are generally consulted. sort events are similarly extremely fast, with off-screen updates virtualised and deferred
  • in my tests, a list with 170,000 rows now sorts in about four seconds. my code is still connected to a non-optimised part of the old system, so I hope to improve gains with background cleanup work in coming months. I believe I can make it work at least twice as fast in places, particularly in initialisation
  • multi-column lists are much better about initialising/migrating the selection 'focus' (the little, usually dotted-line-border box that says where keyboard focus is) through programmatic insertions and deletes and sorts
  • column headers now show the up/down 'sort' arrows using native style. everything is a bit more Qt-native and closer to C++ instead of my old custom garbage
  • none of this changes anything to do with single-column lists across the program, which are still using somewhat jank old code. my taglist in particular is an entirely custom object that is neat in some ways but stuck in place by my brittle design. the above rewrite was tricky in a couple of annoying ways but overall very worth doing, so I expect to replicate it elsewhere. another open choice is rewriting the similarly entirely custom thumbnail canvas to a proper Qt widget with a QLayout and such. we'll see how future work goes

  • misc

  • fixed the 'show' part of 'pages->sidebar and preview panels->show/hide sidebar and preview panel', which was busted last week in the page relayout cleanup
  • I think I fixed the frame of flicker (usually a moment of page-wide autocomplete input) you would sometimes get when clicking a 'show these files' popup message files button
  • fixed the new shimmie parser (by adding a simpler dupe and wangling the example urls around) to correctly parse r34h tags
  • I think I may have fixed some deadlocks and/or mega-pauses in the manage tag parents/siblings dialogs when entering pairs causes a dialog (a yes/no confirmation, or the 'enter a reason' input) to pop up
  • I think I have fixed the 'switch between fullscreen borderless and regular framed window' command when set to the 'media_viewer' shortcut set. some command-processing stuff wasn't wired up fully after I cleared out some old hacks a while ago
  • the manage tag parents dialog has some less janky layout as it is expanded/shrunk
  • if similar files search tree maintenance fails to regenerate a branch, the user is now told to try running the full regen
  • the full similar files search tree regen now filters out nodes with invalid phashes (i.e. due to database damage), deleting those nodes fully and printing all pertinent info to the log, and tells the user what to do next
  • you can now regen the similar files search tree on an empty database without error, lol
  • while I was poking around lists, I fixed a bit of bad error handling when you try to import a broken serialised data png to a multi-column list

  • client api

  • the `/get_files/search_files` command now supports `include_current_tags` and `include_pending_tags`, mirroring the buttons on the normal search interface (issue #1577)
  • updated the help and unit tests to check these new params
  • client api version is now 69

• version 587

  • all misc this week

  • I made a second stupid typo last week. it raised an error when trying to open the 'manage tag display and search' dialog! it was fixed thanks to a user
  • the current local file domains of a file (e.g. 'my files') are now simply listed in the top-right hover window, above any remote locations or URLs. I think I'm going to make these checkboxes or something in future so we can have one-click file migrations
  • if you set up a _share->export files_ job and one of the internal files is actually missing, the error message now tells you to go check for missing files using the database file maintenance stuff
  • if an export files job that is set to delete internal files breaks half way through, the routine now makes sure only to delete what was actually successful
  • subscriptions now catch program shutdown signals better. previously, this was being handled as an unknown error and delay times and error texts were being set. it now just closes cleanly, no worries
  • the command palette should now match case-insensitively
  • I _may_ have fixed a false-positive delete-lock report ('could not delete files xyz because of delete lock') that can happen in the duplicate filter. also, the 'unable to delete file' popup that happens in this case now quietly prints the current stack to log, which I would be interested in seeing
  • I believe I have fixed several of the false-positive 'hey it looks like you edited this parser, are you sure you want to cancel?' confirmations in the edit parser dialog
  • the automatic datestring parsing routine should now be more resilient against english datestrings when the locale differs significantly (it seems if the locale requires a 24-hour clock, it may be a problem for AM/PM time strings)
  • cleaned up some ancient-and-terrible sash-sizing code that manages the three resizable panels of each media results page. hopefully I fixed an issue in Docker and other places where the media page could spawn with a 0-pixel-wide thumbnail panel
  • fixed a weird/stupid bug with the new scanbar that would sometimes start giving errors on media transitions because it couldn't find its media parent
  • improved how a core UI job waits on the database to be free. it now uses just a little less CPU/fewer thread switches
  • improved how that same UI job waits on the pubsub system to be free, same deal
  • since they reversed the API click-through requirement, removed the 8chan TOS click-through login script from the defaults. existing users will see it set to non-active. 8chan thread watching should work out the box again

  • new list stuff

  • I worked on a new multi-column list class that uses a more intelligent data model. I basically finished it, but I will not launch it yet--it needs a bunch more testing and debugging
  • as a side thing, a variety of list display update calls, even on the old list, are now a little faster

• version 586

  • faster sibling/parent fetching

  • for a while, some users have had extremely slow selective sibling/parent fetching, usually manifesting in sibling/parent display calculation or autocomplete results decoration. with last week's new sibling/parent async dialogs, the problem was suddenly exposed further. thankfully, this situation was a useful testbed, and I have made multiple updates that I believe should remove much if not all of the unreasonable megalag. if you saw 30 second delays in the new sibling/parent dialogs, let me know how this all works for you. the ideal is that simple stuff takes 50ms, and something that behind the scenes might have 14,000 rows (stuff like 'gender:female' in parents can sprawl like this), should be no more than a couple of seconds on first fetch, and much faster thereafter
  • fixed up a bad preload routine in the new sibling/parents dialogs that was doing busy wait and eating up bunch of extra CPU
  • simplified the main sibling/parent chain-following search
  • removed all the UNION queries from the sibling and parent modules; maybe I'll reintroduce it one day, but it doesn't really save much time and can limit search cleverness by making the query planner go bananas
  • further optimised the recursive loop of this search, particularly for parents which has to do some additional sibling ideal lookup stuff to join chains coupled by sibling relations
  • overhauled the tag parents/siblings storage tables from the old two-table combined format to dynamic sub-tables separated by both service_id and status. this makes parent and sibling storage a little more spammy but also significantly smaller and more simple, and it ensures search code is always working on clean, efficient, and fast indices, which means no more crazy search variability no matter how we work with these things. as a side benefit, I relaxed the logic so the siblings storage is now capable of storing more 'conflicting' pairs, no longer enforcing an old overly optimistic 1->n rule (which was probably the cause of some 'I see a different loop to you, how do we debug this?' frustration amongst PTR users comparing siblings). **if you sync with the PTR, the database update to v586 will take a few seconds this week**
  • there may still be a single slow-the-first-time query for parents in a PTR-syncing client, simply because certain joiner tags like 'gender:female' merge many groups together. I am considering what to do here, so let's see how it goes
  • plugged a hole in the 'fetch relevant sibling/parent pairs' routine where if you triggered two searches at the same time with overlapping tags (e.g. let's say things were working super slow), the second routine was not waiting correctly for the results and the main EnterPairs method was raising a 'hey, this should not have happened' message

  • misc

  • fixed an issue in the media scanbar where if you had it set to hide completely when the mouse is not over it, then if the media was paused while the scanbar was hidden, the scanbar would unhide in a blank state until you clicked it. further, the anti-show/hide-flicker tech is improved here
  • when you open up a tag search page from the media viewer's tag list (e.g. by middle-clicking a tag), the original context's file domain is now preserved. if you open a media viewer on 'my files', then new search pages from the taglist will now be in 'my files' (it was previously defaulting to the safe backstop of 'all my files')
  • the client now forces a full tag presentation refresh when deleting a service or resetting a tag respository's processing. this should clear up some ghost tags we were seeing here without having to restart
  • the master decoding call used by the parsing system (which does 'convert this raw I/O input to nice unicode text') will now implicitly trust encoding provided by the network engine if that encoding is exlpcitly set in the response (previously it would defer to `chardet` if that was more confident), and if the given document is encoded incorrectly, it will replace bad characters with special question marks
  • when an import options button handles only one options type (e.g. the tag import options button in edit subscription query panel, where it also only does 'additional tags' stuff, or the file import buttons in _options->importing_), the button now previews what it does in its label. the way these summary statements is produced (and, more generally, used in the button's tooltips) is also tightened up--there is less newline spam, and smaller changes will collect into a single line
  • because of some remaining display bugs, if your Qt's default style would be the new 'windows11' (which is true for Win Qt 6.7.x), I am saying 'no' and switching it back to 'windowsvista'
  • I removed a 'do not allow an import folder to run for more than an hour' timer. this was an undocumented backstop hack and was messing with 'do not run regularly' import folders that operate on 100,000+ file mega folders. if you want a gigantic import job, you got it
  • silenced some spammy network reporting--the main file and gallery import objects were printing tracebacks to the log on many failure states, which in some unusual SSL/Connection errors was resulting in a whole lot of html garbage being dumped to the log
  • improved the error message when an audio file's duration cannot be determined
  • tweaked the 'help my db is broke.txt' document
  • fixed up some weird tag application logic: the client db and the tags manager object now agree that you can, through programmatic means, petition content that does not yet exist (e.g. to insert deleted rows from an external source), and thus if you wish to _pend_ content, we need to check for conflicting pre-existing _petitioned_ content, and _vice versa_. the manage tags dialog similarly understands this, but it won't offer the 'petition' action when things do not yet exist because this is a bit technical and best left to programmatic editing like the Client API or migrate tags window. it was previously possible to create a situation where a file had both pending and petitioned data that did not yet exist (`tag (+1) (-1)`, lol)--this should no longer be possible. if you got into this situation and want to clean it up, try doing a search for 'system:has tags' on just 'include pending tags', and then ctrl+a->F3 your results and then ctrl+a the taglist and hit enter on it--you should be given an option to 'undo petition on x tags' and clear it all up in one go

  • noneable defaults

  • all of the 'noneable' (nullable) integer widgets (where you have an editable number with a 'no limit' checkbox beside it) now initialise with an appropriate default value in the integer box, even if they otherwise initialise in the 'None' state. previously, these would usually sit at '1' on the number side, when starting at None, meaning you'd have to guess an appropriate number when switching from None to something concrete. all the noneable integers in the options dialog now initialise with their respective options default
  • similarly, most of the noneable text input boxes now initialise with a suggested value in the text box even if the initial value for that dialog or whatever is the 'None' checkbox ticked
  • and all of the nullable bytes widgets (a number-of-bytes value and then 'no limit' checkbox) similarly now initialise with a default value. they kind of already did, but it is better formalised now
  • dejanked some nullable int widget code design. the ones that have two dimensions are now their own class

  • client api

  • thanks to a user, `/get_files/render` has new parameters that let you now ask for a png/jpeg/webp rather than just png, at a certain quality, and a certain resolution
  • added the 'sort by pixel hash hex and blurhash' sort_type definitions to the help for `/get_files/search_files` and noted that you can asc/desc these too
  • `/add_files/add_file` now accepts a 'file domain' to set a custom import destination (just like in file import options). obviously you can only set local file domains here
  • `/add_urls/add_url` also now accepts a 'file domain', same deal. it will select/create a new url downloader page with non-default file import options set with that import destination
  • updated the help and unit tests to reflect the above
  • added `/add_urls/migrate_files` to copy files to new local file domains (essentially doing _files->add to_ from the thumbnail menu)
  • with (I think) all multiple local file service capabilities added to the Client API, issue #251 is finally ticked off
  • client api version is now 68

• version 585

  • the new asynchronous siblings and parent dialogs

  • the `tags-$gt;manage tag siblings/parents` dialogs now load quickly. rather than fetching all known pairs on every open, they now only load pertinent pairs as they are needed. if you type in tag A in the left or right side, all the pairs that involve A directly or point to a pair that involves A directly or indirectly are loaded in the background (usually so fast it seems instant). the dialog can still do 'ah, that would cause a conflict, what do you want to do?' logic, but it only fetches what it needs
  • the main edit operations in this dialog are now 'asynchronous', which means there is actually a short delay between the action firing and the UI updating. most of the time it is so fast it isn't noticeable, and in general because of other cleanup it tends to be faster about everything it does
  • the dialogs now have a sticky workspace 'memory'. when you type tags in, the dialog still shows the related rows as normal, but now it does not clear those rows away once you actually enter those new pairs. the 'workspace' shows anything related to anything you have typed until you hit the new 'wipe workspace' button, which will reset back to a blank view. I hope this makes it less frustrating to work on a large group--it now stays in view the whole time, rather than the 'current' stuff jumping in and out of view vs the pending/petitioned as you type and submit stuff. the 'wipe workspace' button also has the current workspace tags in its tooltip
  • the 'show all pairs' checkbox remains. it may well take twenty seconds to load up the hundreds of thousands of pairs from the PTR, but you can do it
  • also added is a 'show pending and petitioned groups', which will load up anything waiting to be uploaded to a tag repository, and all related pairs
  • when a user with 'modify siblings/parents' adds a pair, the auto-assigned 'reason' is now "Entered by a janitor.' (previously it was the enigmatic "admin")
  • some misc layout improvements aross the board. the green/red text at the top is compressed; the 'num pairs' now shows the current number of pairs count; there are more rows for the pairs list, fewer for the input list; and the pairs list eats up all new expand space
  • a great amount of misc code cleanup in all these panels and systems, and most of the logic is shared between both sibling and parent dialogs. a lot of janky old stuff is cleared up!
  • these dialogs are better about showing invalid, duplicated, or loop-causing pairs. the idea is to show you everything as-is in storage so you can better directly edit problems out (previously, I am pretty sure it was sometimes collapsing stuff and obscuring problems)
  • the 'manage tag parents' dialog now auto-petitions new loops when entering pairs (it was just siblings before)
  • this tech now works on multiple potential loops, rather than just the first
  • the 'manage tag parents' dialog now detects pre-existing loops in the database record and warns about this when trying to enter pairs that join the loop (it was just siblings before)
  • this tech works better and now detects multiple loops, including completely invalid records that nonetheless exist (e.g. `a->b, a->c` siblings that point to more than one locations), and when it reports them, it now reports them all in one dialog, and it shows the actual `a->b->c->d` route that forms the loop
  • a bad final 'do not allow loop-inputs' backstop check in the main pair-add routine is removed--it was not helping

  • misc

  • hitting escape on any taglist will now deselect all tags
  • added 'Do not allow mouse media drag-panning when the media has duration' to the _options->media viewer_ page. if you often misclick and pan when scrubbing through videos, try it out!
  • the media viewer's top hover window no longer shows every 'added-to' time for all the local file services; it was spammy, so it now just says 'imported: (time)'. the related 'hide uninteresting import time' option is retired. I also removed the 'archived: (time)' label, so this is now pretty much just 'imported, modified'. if I bring detailed times back to the file summary, it'll be part of a more flexible system. note that all these timestamps are still available in the media top-row flyout menu
  • the file log and gallery log now copy their urls/sources on a ctrl+c hit. also, the 'copy' right-click commands here also no longer unhelpfully double-newline-separates rows
  • a `StringConverter` edit panel now throws up a yes/no confirmation if you try to ok on a regex substitution that seems to match a group in the pattern but has an empty string in the 'replacement' box
  • updated the 'test' versions of OpenCV (4.10.0.84), Pyside6 (6.7.2), and python-mpv (1.0.7). I'll be testing these myself, and devving with them, mostly to iron out some Qt 6.7.x stuff we've seen, and then put out a future release with them
  • added a note to the default_mpv.conf to say 'try commenting out the audio normalisation line if you get mpv problems and are on Arch'
  • added different example launch paths to the 'external programs' options panel depending on the current OS
  • added a note about running with `QT_QPA_PLATFORM=xcb` on Wayland to the install help
  • refactored the `ClientGUIFileSeedCache` and `ClientGUIGallerySeedLog` files, which do the file and gallery log panels, up to the 'gui.importing' module
  • thanks to a user, added a new darkmode 'Nord' stylesheet

  • fixes

  • fixed 'scrub invalidity' in the manage logins dialog--sorry, it was a stupid typo from the recent multiple-column list rework. also, this button is now only enabled if the login script is active
  • fixed a bug opening the 'migrate files' dialog when you have no files!
  • I force-added `Accept-Language: en-US,en;q=0.5` to the client's default http headers for pixiv.net. this appears to get the API to give us English tags again. let me know if this completely screws anything up
  • updated the 'do we have enough disk space to do this transaction?' test to check for double the destination disk amount. thanks to the user who helped navigate this--regardless of temp dir work, when you do a vacuum or other gigantic single transaction, there is a very brief period as the transaction commits when either the stuffed WAL journal or (for a vacuum) cloned db file exists at the same time in the same folder as the original db file. I also updated the text in the 'review vacuum data' window to talk about this a bit. good luck vacuuming your client.mappings.db file bros
  • improved the error handling when a sidecar import fails--it now says the original file path in the report
  • improved failure-recovery of unicode decoding (usually used in webpage parsing) when the given text includes errors and the encoding is `ISO-8859-1` (or the encoding is unparseable and `requests` falls back to it) and/or if `chardet` is not available
  • I hacked the menubar padding back to something sensible on the new 'windows11' style int Qt 6.7.x. for whatever reason, this new style adds about 15px of padding/margin to each menubar menu button. I am aware the collect-by combobox is still busted in this style--let me know if you spot anything else! btw switching from 'windows11' to 'windowsvista' seems to make all the menubar menus transparent, let's go
  • improved the layout of the 'edit client api access key permissions' panel. it wasn't vertically expanding before
  • fixed up some keypress handling in taglists. some stuff that was being swallowed or promoted unintentionally is fixed
  • thanks to a user, fixed a weird bug in the 'repair missing file storage locations' boot repair dialog where it would always say you only had missing thumbs
  • also thanks to that user, the 'repair missing file storage locations' dialog now checks `client_files` and `thumbnails` subdirectories when trying to auto-discover with the 'add a possibly correct location' action

  • some hash-sorting stuff

  • _you can probably ignore this section, don't worry about it_
  • you can now sort by blurhash. this works at the database level too, when mixed with system:limit
  • when sorting by pixel hash, a file search with system:limit now pre-sorts by pixel hash before the limit clips the resultset
  • when sorting by pixel hash or blurhash, the files with no such hash (e.g. audio files) are now always put at the end
  • searching many tens of thousands of files and sorting by hash, pixel hash, or blurhash is now just a tiny bit faster

  • client api

  • the new `/manage_services/get_pending_counts` command now includes the 'Services Object' in its response
  • the client api version is now 67

• version 584

  • misc

  • fixed a logical hole in the recent 'is this URL that is saying (deleted/already in db) trustworthy, or does it have weird mappings to other files?' pre-download check that was causing Pixiv, Kemono, and Twitter and any other multiple-URL Post URL Class to, on re-encountering the URL in a downloader, classify the underlying file URL as untrustworthy and re-download the files(l!!)
  • the 'copy all' and paste buttons in the manage known urls dialog are replaced with icon buttons, and the copy button now copies the current selection if there is one
  • the newish Regex input widget (the one that goes green/red based on current text validity) now propagates an Enter key into a dialog ok event when appropriate
  • when you ctrl+double-click a taglist, the program now ensures the item under the mouse is selected before firing off the double-click nega-activation event. this is slightly awkward, but I hope it smoothes out the awkward moment where you want to invert a selection of tags but doing a normal ctrl+double-click on them causes the one of them to be deselected and then messes up the selection
  • regex URL searches are now always the last job to run in a file query. if you mix in any other predicate like filesize or just some tag, your regex URL searches should run massively massively faster
  • improved some boot error handling when Qt fails to import
  • fixed the whack alignment of the 'filename'/'first directory'/etc.. checkbox-and-text-edit widgets in the filename tagging panel, and set 'namespace' placeholder text
  • force-selecting a null value in a 'select one from this list of things' dialog should no longer raise errors
  • thanks to a user, the new shimmie parser gets tags in a simpler, more reliable way

  • client api

  • added a new permission, `Commit Pending` (12), which allows you to see and commit pending content for each service
  • added `/manage_services/get_pending_counts`, which basically returns the content of the client's 'pending' menu
  • added `/manage_services/commit_pending`, which fires those commands off
  • added `/manage_services/forget_pending`, which does the same 'forget' command on that menu
  • added `/manage_file_relationships/remove_potentials`, which clears any known potential pairs off the given files
  • the `/manage_pages/get_pages` and `/manage_pages/get_page_info` commands now return `is_media_page` boolean, which is a simple shorthand for 'not a page of pages'
  • added the above to the Client API help
  • wrote unit tests covering the above
  • the client api version is now 66

  • boring cleanup

  • fixed up how some lists deliver their underlying data to various methods
  • `CallBlockingToQt` no longer spams encountered errors to the log--proper error handling should (and does now) occur elsewhere
  • the way the initial focus is set on system predicate flesh-out panels (when you double-click on like 'system:dimensions' and get a bunch of sub-panels) is more sane. should be, fairly reliably, on the first editable panel's ok button. I want it to be on an editable widget in the panel in future, I think, but I need to do some behind the scenes stuff to make this work in a nicer way
  • pulled some stuff out of `HydrusData`, mostly to `HydrusNumbers`, `HydrusLists`, and the new `HydrusProcess`, mostly for decoupling purposes
  • renamed some `ConvertXToY` stuff to just `XToY`

• version 583

  • new

  • added a 'command palette' options page. the ability to show menubar and media actions is now set here with a couple of checkboxes (previously these were hidden behind advanced mode)
  • you can also now search for 'page of pages' with the command palette. this is turned on in the same new options page
  • the new sidecar test panel now only shows its notebook tabs if there is more than one source. the tab labels, when shown, are now '1st' ... '2nd' ordinal strings respective to the source list on the left
  • added some explanation text, a button to the help docs, and a bit of better layout to the sidecars UI
  • the file import report mode now writes the 'nice human description' for the file import to the log. this will expose the source URL or local file path for better context than just the random temp path
  • in the 'manage times' dialog, you can now set the same timestamp (including cascading timestamps) to multiple domains or file services at once. the logic is a little delicate where some files are in one domain in your selection and others are only in another, but I think I got it working ok. if you have a complicated setup, let me know how you get on!
  • the 'Dark Blue' stylesheets and the new 'Purple' stylesheet have updated custom colours

  • fixes

  • fixed a typo in the 'fetch a mappings petition' server code
  • when the client is called to render an image that was just this moment removed from the file store, the image renderer should now give you a nicer error image with an appropriate messare rather than throwing an error popup
  • hydrus no longer applies EXIF Orientation (rotation) to PNG files, which it recently started doing automatically when we started scanning PNGs for EXIF. EXIF is not well-defined and supported for PNG, and if an Orientation row exists, it is likely a false positive mistake of some encoder. hydrus will show this data but it will not apply it, and the metadata review UI now also notes this
  • fixed the 'has exif?' file maintenance job, which I think was false-negativing!

  • code and help

  • the `psutil` library is now technically optional. it is still needed for a bunch of normal operations like 'is the client currently running?' and 'how much free space is there on this drive?', but if it is missing the client will now boot and try to muddle through anyway
  • did more multi-column list 'select, sort, and scroll' tech for: the url class parameters list; the url class links list; the external launch paths list; the tag suggestion related tags namespace lists; import folder filename tagging options; manage custom network context headers; the string to string match widget list; the string match to string match widget list; the edit subscription queries list; and more of the edit subscriptions subscription list, including the merge and separate actions
  • updated the QuickSync help a little, clarifying it is only useful for _new, empty_ clients

• version 582

  • fixes

  • fixed an issue where setting a file 'collect' was not automatically sorting the collected objects internally properly. normally when you collect, each collected object is supposed to be sorted internally by filesize or namespace or whatever--this is working again
  • fixed a weird internal error state in the import folders manager where it could get confused about and throw an error regarding the import folders' next work times if an internal update notification occured during an import folder working
  • fixed a typo error with the shortcut set 'special duplicate' button
  • fixed pasting new query texts into the manage subscriptions dialog when one of the pasted texts resurrects a DEAD query. my new summary generation text was handling the DEAD report wrong!

  • misc

  • the advanced 'all deleted files' service, which is mostly just used for behind the scenes caching calculations, is renamed to 'deleted from anywhere'. the related 'regen->all deleted files' database command is also moved to 'check and repair->sync combined deleted files'
  • the edit tag filter panel's 'load' button now shows all the current tag repositories' tag filters
  • when you hit ctrl-enter on some tags (or otherwise trigger a linked remove+add action) in an active search list (e.g. top-left on a search page), which causes those tags to invert and thus sometimes sorted to a different position, the current selection now propagates through the inversion, with the keyboard focus moved to the post-topmost item. so, you can now basically hit ctrl+enter twice for a no-op
  • fixed the paste button in the new 'purge tags' dialog
  • thanks to a user, we have a new 'Purple' stylesheet
  • I tweaked the some default stylesheet colours and think I fixed the display of the 'valid/invalid' controls you sometimes see (for instance in the new regex input, which goes green/red) for dark mode stylesheets that don't define colours for these. previously, the dark mode text, usually a light grey, was being washed out by the default green

  • custom colours in QSS

  • you can now set the _options->colours_ colours in a QSS stylesheet! if you are a stylesheet maker, check the default_hydrus.qss file to see how it works--it is the same deal as the animation scanbar previously
  • the options in _options->colours_ remain, but they are now wrapped in a 'overwrite your stylesheet with these colours' checkbox _for now_. existing users are going to be set to 'yes overwrite', so nothing will suddenly change, but new users are going to default to using whatever the current QSS says. in future, I may collapse the light/darkmode distinction into one option set; I may morph it into a "colour highly rated files' thumbnail borders gold" dynamic options system; I may simply delete the whole thing and replace it with in-client QSS editing or something. not sure, so let's see how it goes and how Qt 7's darkmode stuff turns out.
  • I have pasted the hydrus default darkmode colours into all the other stylesheets that come with the program, so new users selecting a darkmode style are going to get something reasonable out of the box rather than the previous ugly clash. the users who made the original stylesheets are welcome to figure out better colours and send them in
  • if you are not set to override with the custom colours in _options->colours_, then hitting _help->darkmode_ now gives you a popup telling you what is going on

  • sidecar UI

  • the four 'edit sidecars' panels (under manual imports, import folders, manual exports, and export folders), which use paths and media as sources respectively, now have test panels to review the current sidecar route you have set up. they use up to 25 rows of example file paths/media from the actual thing you are working on. it provides a live update of what the sources you set up will load, so you know you have the json parse or .txt separator set up correct
  • for the export folders case, there is a button in the panel 'edit export folders' panel to populate the text context, under your control, since this involves a potentially slow file search
  • there is more to do here. I would like better test panels in the sub-dialogs and I'd like to collapse the related 'eight-nested-dialogs-deep' problem, and in the string processing and parsing UI more generally, but I'm happy with this step forward. let me know where it goes wrong!

  • advanced autocomplete logic fixes

  • when you enter a wildcard into a Read tag autocomplete, it no longer always delivers the 'always autocompleting' version. so, if you enter `sa*s`, it will suggest `sa*s (wildcard search)` and perhaps `sa*s (any namespace)`, but it will no longer suggest the `sa*s*` variants until you, obviously, actually type that trailing asterisk yourself. I intermittently had no idea what the hell I was doing when I originally developed this stuff
  • the 'unnamespaced input gives `(any namespace)` wildcard results' tag display option is now correctly negatively enforced when entering unnamespaced wildcards. previously it was always adding them, and sometimes inserting them at the top of the list. the `(any namespace)` variant is now always below the unnamespaced when both are present
  • fixed up a bunch of jank unit tests that were testing this badly

• version 581

  • misc

  • thanks to a user, we have a much improved shimmie parser, for both file and gallery urls, that fetches md5 better, improves gallery navigation, stops grabbing bad urls and related tags by accident, and can handle namespaces for those shimmies that use them. for our purposes, this improves r34h and r34@paheal downloaders by default
  • thanks to a user, we have a new 'Dark Blue 1.1' styesheet with some improvements. the recommendation is: check the different scrollbar styling to see if you prefer the old version
  • timedelta widgets now enforce their minimum time on focus-out rather than value change. if it wants at least 20 minutes, you can now type in '5...' in the minutes column without it going nuts. let me know if you discover a way to out-fox the focus-out detection!
  • added a checkbox to file import options to govern whether 'import destinations' and 'archive all imports' apply to 'already in db' files. this turns on/off the logic that I made more reliable last week. default is that they do
  • added 'do sleep check' to _options->system_ to try some things out on systems that often false-positive this check
  • the 'review current network jobs' multi-column list has a new right-click menu to show a bit more debug info about each job: each of its network contexts, how the bandwidth is on each context, if the domain is ok, if it is waiting on a connection error, if it is waiting on serverside bandwidth, if it obey bandwidth, and if its tokens are ok. if you have been working with me on gallery jobs that just sit on 'starting soon', please check it out and let me know what you see. also, 'review current network jobs' is duplicated to the help->debug menu. I forgot where it was, so let's have it in both places
  • on the filename-import tagging panel, the filename and directory checkbox-and-text-edit widgets no longer emit a (sometimes laggy) update signal when typing when the checkbox is unchecked

  • janitor stuff

  • if you are a repository janitor, right-clicking on any tag shows a new 'admin' menu
  • if you have 'change options' permission, you will see 'block x'/'re-allow x' to let you quickly see if tags are blocked and then edit the repository tag filter respectively
  • if you have 'mappings petition resolution' permission, you can 'purge' the selected tags, which will deleted them from the service entirely. this launches a review window that previews the job and allows adding of more tags using the standard autocomplete interface. when 'fired off', it launches a tag migration job to queue up the full petition/delete upload
  • this new 'purge' window is also available from the normal 'administrate services' menu in the main gui
  • also under the 'administrate services' is a new 'purge tag filter' command, which applies the existing repository tag filter to its own mappings store, retroactively syncing you to it

  • tag filters and migration

  • I wrote a database routine that quickly converts a hydrus tag filter into the list of tags within a file and tag search context. this tech will have a variety of uses in the genre of 'hey please delete/fetch/check all these tags'
  • to start with, it is now plugged into the tag migration system, so when you set up, say, an 'all known files' tag migration that only looks for a namespace or a bunch of single tags, the 'setup' phase is now massively, massively faster (previously, with something like the PTR, this would be scanning through tens of millions of files for minutes; now it just targets the 50k or whatever using existing tag search tech usually within less than a second)
  • cleaned (KISSed) and reworked the tag filter logic a bit--it can now, underlyingly, handle 'no namespaced tags, except for creator:anything, but still allowing creator:blah'
  • optimised how tag filters do 'apply unnamespaced rules to namespaced tags' (which happens in some blacklists that want to be expansive)
  • improved how the tag filter describes itself in many cases. it should make more grammatical sense and repeat itself less now (e.g. no more 'all tags and namespaced tags and unnamespaced tags' rubbish)
  • improved how some tag filter rules are handled across the program, including fixing some edge-case false-positive namespace-rule detection
  • deleted some ancient and no longer used tag filtering code

  • boring multi-column list stuff

  • did more 'select, sort, and scroll' code cleanup in my multi-column lists, specifically: manage import folders; manage export folders; the string-to-string dict list; edit ngug; edit downloader display (both gugs and url classes, and with a one-shot show/hide choice on a multi-selection rather than asking for each in turn); the special 'duplicate' command of edit shortcut set; and the string converter conversions list (including better select logic on move up/down)
  • in keeping with the new general policy of 'when you edit a multi-column list, you just edit one row', the various 'edit' buttons under these lists across the program are now generally only enabled when you have one row selected
  • the new 'select, sort, and scroll to new item when a human adds it' tech now _deselects_ the previous selection. let me know if this screws up anywhere (maybe in a hacky multi-add somewhere it'll only select the last added?)
  • the aggravating 'clear the focus of the list on most changes bro' jank seems to be fixed--it was a dumb legacy thing
  • whenever the multi-column list does its new 'scroll-to' action, it now takes focus to better highlight where we are (rather than stay, for instance, leaving focus on the 'add' button you just clicked)

  • other boring stuff

  • worked a little more on a routine that collapses an arbitrary list of strings to a human-presentable summary and replaced the hardcoded hacky version that presents the 'paste queries' result in the 'edit subscription' panel with it
  • wrote a similar new routine to collapse an arbitrary list of strings to a single-line summary, appropriate for menu labels and such
  • fixed a layout issue in the 'manage downloader display' dialog that caused the 'edit' button on the 'media viewer urls' side to not show, lmaooooooo
  • ephemeral 'watcher' and 'gallery' network contexts now describe themselves with a nicer string
  • decoupled how some service admin stuff works behind the scenes to make it easier to launch this stuff from different UI widgets
  • refactored `ToHumanInt` and the `ToPrettyOrdinalString` guys to a new `HydrusNumbers.py` file
  • fixed some bad Client API documentation for the params in `/get_files/search_files`

• version 580

  • misc

  • I _may_, and a very hesitant _may_, have fixed the program hanging after minimising to system tray from the close button. thanks to the user who pinned down that it was the close button doing this rather than the other ways to minimise to system tray. if you have had trouble with minimising to the system tray, please try again when it is convenient and let me know how you get on. please also note which exact command, whether it was the file menu, system tray icon menu, minimise button, or close button, that you hit to trigger the minimise event that ultimately would not restore correctly
  • the taglist right-click menu now has a _maintenance->regenerate tag display_ command, which is basically the 'regenerate mappings storage cache' command in the database menu, but limited just to your selection. this _should_, with luck, fix incorrect autocomplete counts or sibling/parent presentation for any tags you see that are weird. I've wanted this for years, since the whole-cache regen is so large that it is essentially impossible to run on the PTR, but now we can debug individual tag presentation problems a lot easier!
  • fixed an issue where read-only import files would not delete from the temp dir after import, despite, if desired, successfully deleting from their original locations. it turns out the read-only property was being copied to the temp path for import, and the 'I'm done with the temp file, delete it' routine, unlike the normal file delete, wasn't checking for and undoing read-only status. note this was also screwing with the 'delete the hydrus temp dir on shutdown' routine, so if you do a lot of unusual/misc hard drive imports, feel free to shut your client down, check your temp folder (hit _help->about_ to find it), and delete anything called hydrusXXXXXXXX
  • the new 'eye' icon in the media viewer now has 'apply image ICC Profile colour adjustments', which will flip on/off the fairly newish checkbox added to _options->media playback_. it updates the image live!
  • added a shortcut for the 'flip apply image ICC Profile colour adjustments' to the 'media viewer' set! if you are big into this stuff and also do duplicate filtering, set it up and let me know how it goes
  • important but subtle file import options fix: when you set a file to import to a specific destination in file import options, or you say to archive all imports, this is supposed to work even when the file is 'already in db'. this was not working when 'already in db' was caused by a 'url/hash recognised' result in the downloader system. I have fixed this; it now works for 'already in db' for url/hash/file recognised states. thank you to the user who noticed this and did the debug legwork to figure out what was going on
  • import _file logs_ now have a menu item 'search for URLs', which does the same as the recent 'urls' media right-click menu command, opening a search page for any files that share these URLs
  • added a shortcut command 'reload the current qss stylesheet' to the 'main gui' shortcut set. moreover, the 'reload current ui session' entry in the debug menu, which was just above this before, is renamed to 'close and reload current ui session' because of common misclicks
  • the options panel uses less CPU on ok/cancel to set/reset style as needed. same deal with the old hack that makes the colour-picker work--it'll now be more efficient about setting/resetting style
  • fixed a stupid list/tuple type error when trying to edit the 'frame locations' in options->gui. this was from an accident during the selection/scroll rewrites last week
  • generally improved the reliability of the multi-column list against the above bug in its various forms
  • added a simple click-through login script to fix recent changes to the 8chan.moe TOS filter, which broke the respective watcher. all users get this and it should just work out of the box
  • thanks to a user, the default danbooru parsers are fixed to fetch md5 hash correctly
  • some misc tooltip and description fixes
  • improved some media result testing stuff

  • client api

  • the `/add_tags/add_tags` command has two new parameters, `override_previously_deleted_mappings`, and `create_new_deleted_mappings`, both True by default (which was also previous behaviour). turning either off allows you to, respectively, not force-add a tag mapping when it has been previously deleted (like how the gallery downloader works) and not force-delete (and thus make a 'delete' record) when deleting a tag mapping unless it already exists
  • updated the Client API help to talk about these
  • added some unit tests to test these
  • the client api is now version 65

• version 579

  • some url-checking logic

  • the 'during URL check, check for neighbour-spam?' checkbox in _file import options_ has some sophisticated new logic. check the issue for a longer explanation, but long story short is if you have two different booru URLs that share the same source URL (with one or both simply being incorrect e.g. both point to the same 'clean' source, even though one is 'messy'), then that bad source URL will no longer cause the second booru import job to get 'already in db'. it now recognises this is an untrustworthy mapping and goes ahead with the download, just as you actually want. once the file is imported, it is still able, as normal, to quickly recognise the true positive 'already in db' result, so I believe have successfully plugged a logical hole here without affecting normal good operation! (issue #1563)
  • the 'associate source urls' option in file import options is more careful about the above logic. source urls are now definitely not included in the pre-import file url checks if this option is off

  • some regex quality of life

  • regex input text boxes have been given a pass. the regex 'help' links are folded into the button, the links are updated to something newer (one of the older ones seems to have died), the button is now put aside the input and labelled `.*`, the menu is a little neater, and the input has placeholder text and now shows green/red (valid/invalid in the stylesheet) depending on whether the current regex text compiles ok. just a nicer widget overall
  • this widget is now in filename tagging, the String Match panel regex match, the String Converter panel regex step, and the 'regex favourites' options panel, which I was surprised to learn the existence of
  • the regex menu for the String Converter regex step also now shows how to do grouping in python regex. I hadn't experimented with this properly in python, but I learned this past week that this thing can handle `(...) -> \1` group-replace fine and can do named groups with (?P<name>...) -> \g<name> too!
  • for convenience, when editing a String Match, if you flick from 'any' to 'fixed' or 'regex', it now puts whatever was in your example text beforehand as the new value for the fixed text or regex

  • list selecting and scrolling

  • I added some new scroll-to tech to my multi-column lists
  • pasting a URL into the 'edit URL Classes' dialog's test input now selects and scrolls to the matching URL Class
  • the following lists should all have better list sort/select preservation, and will now scroll to and maintain visibility, on various edit/add events: edit url classes, edit gugs, edit parsers, edit shortcut sets, edit shortcut set, the options dialog frame locations, the options dialog media viewer options, manage services, manage account types, manage logins, manage login scripts, edit login script, and some weird legacy stuff. lots more to do in future
  • when you 'add from defaults' for many lists, it will now try and scroll to what was just added. may not be perfect!
  • same deal with 'import' buttons. it will now try and scroll to what you import!
  • I am also moving to 'when you edit, you only edit one row at a time'. in general, when I have written list edit functions, I write them to edit each row of a multi-selection in turn with a new dialog, but: this is not used very much, can be confusing/annoying to the user, and increases code complexity, so I am undoing it. as I continue to work here, if you have a multi-selection, an edit call will increasingly just edit the top selected row. maybe in this case I'll reduce the selection, maybe I'll add some different way to do multi-edit again, let me know what you think

  • misc

  • import folders now work in a far more efficient way. previously, the client loaded import folders every three minutes to see which were ready to run; now, it loads them once on startup or change and then consults each folder to determine how long to wait until loading it again. it isn't perfect yet, but this ancient, terrible code from back when 100 files was a lot is now far more efficient. users with large import folders may notice less background lag, let me know how you get on. thanks to the users who spotted this--there's doubtless more out there
  • to help muscle memory, the 'undo' menu is now disabled when there is nothing for it to hold, not invisible. same deal for the 'pending' menu, although this will still hide if you have no services to pend to (ipfs, hydrus repositories). see how this feels, maybe I'll add options for it
  • the new 'is this webp animated?' check is now a little faster
  • if your similar file search tree is missing a branch (this can happen after db damage or crash desync during a file import) and a new file import (wanting to add a new leaf) runs into this gap, the database now imports the file successfully and the user gets a popup message telling them to regen their similar files search tree when convenient (rather than raising an error and failing the import)
  • added a FAQ question 'I just imported files from my hard drive collection. How can I get their tags from the boorus?', to talk about my feelings on this technical question and to link to the user guide here: https://wiki.hydrus.network/books/hydrus-manual/page/file-look-up
  • the default bandwidth rules for a hydrus repository are boosted from 512MB a day to 2GB. my worries about a database syncing 'too fast' for maintenance timers to kick in are less critical these days

  • build and cleanup

  • since the recent test 'future build' went without any problems, I am folding its library updates into the normal build. Qt (PySide6) goes from 6.6.0 to 6.6.3.1 for Linux and Windows, there's a newer SQLite dll on Windows, and there's a newer mpv dll on Windows
  • updated all the requirements.txts to specify to not use the brand new numpy 2.0.0, which it seems just released this week and breaks anything that was compiled to work with 1.x.x. if you tried to set up a new venv in the past few days and got weird numpy errors, please rebuild your venv in v579, it should work again
  • thanks to a user, the Docker build's `requests` 'no_proxy' patch is fixed for python >3.10
  • cleaned up a ton of `SyntaxWarnings` boot logspam on python >=3.12 due to un-`r`-texted escape sequences like `\s`. thanks to the user who submitted all this, let me know if I missed any
  • cleaned up some regex ui code
  • cleaned up some garbage in the string panel ui code
  • fixed some weird vertical stretch in some single-control dialogs

• version 578

  • animated webp

  • we now have animated webp support! despite many libraries having trouble with this, it turns out that modern PIL can decode and render them. I have figured out a solution using my old native gif renderer, so webps will now play in the program
  • I'm going the same route as gifs and (a)pngs--the program now tracks 'webp' vs 'animated webp' as different filetypes. all your image webps will be queued for a scan on update, and any with animation will become the new type, with num frames and duration, and will be fully viewable in the media viewer
  • I don't know when PIL added this tech, so if you are a source user and haven't rebuilt your venv in a while, this is probably a good time!

  • misc

  • the five new 'draw hover-window text in the background of media viewer' options are now copied to the media viewer itself, under a new 'eye' icon menu button. I'll be hanging more stuff off here, like 'always on top' in future!
  • the 'known urls' media submenu is now just 'urls', and it now has A) the 'manage' command, moved from the manage menu and B) 'open in a new page' for the focused file's specific URLs or 'any of them' (i.e. it opens a new search page with 'system:known url=blah', so if you need to find which files share a URL, it is now just one click
  • fixed the gelbooru 0.2.5 post parser's fetching of multiple source urls. it was not splitting them correctly due to a (recent?) change on gelbooru's end and adding unhelpful `https://gelbooru.com/%7C` gumpf as an additional source url
  • added a 'network report mode (silent)' to the `help->debug` menu, which does everything the network report mode does but with silent logging rather than a million popups. should help with longer-term debugging
  • fixed an issue with fetching gif variable framerate timings in the native renderer
  • added variable framerate tech to the native renderer for all animation types PIL can figure out except apng (previously it could only do it for gif)
  • tightened up some of the file metadata checks. apng is now scanned for exif data, the HEIF types are now scanned for transparency. these jobs are also queued up on update
  • the media viewer's top hover's center buttons (usually inbox/delete stuff) are finally centered correctly, aligned with the text below. apologies for the delay; it took several years of under-waterfall mediation to gather enough chi, but I finally have a beginner's understanding of `QSizePolicy.Expanding`

  • boring cleanup

  • fixed some more long tooltips to wrap into nice newlines
  • converted the 'manage urls' dialog to the decoupled 'edit' paradigm
  • refactored most of the 'scrolling panel' code to a new `gui.panels` module
  • broke up some of the bloated scrolling panel code into smaller files, moved Migrate Tags and Edit Timestamps to new files in `gui.metadata`, and replaced/deleted some old code
  • refactored `ClientGUITime` to `gui.metadata`, `ClientGUILogin` to `gui.networking`

• version 577

  • explorer integration

  • thanks to a user, we have some new OS-file-explorer integration
  • two additional options are added to the "open" menu for Windows users, "in another program" opens the Windows dialog to select which program to use and "properties" opens the Windows file properties dialog for the file
  • the 'media' shortcut set gets the new 'open file properties' and 'open with...' commands to plug into these new features
  • the "open in file browser" media menu command now more reliably selects the file in Windows and is now available for most Linux file managers--full list [here](https://github.com/damonlynch/showinfilemanager#supported-file-managers).
  • the "open files' locations" file import log menu command is similarly more reliable, and can sometimes select multiple files when launched on a selection
  • this requires a new external library, so users who run from source will want to rebuild their venvs this week to get this functionality

  • misc

  • the manage times single-time edit dialog's paste button can now eat any datstring you can think of. try pasting 'yesterday 3am' into it, it'll work!
  • split the increasingly cluttered 'media' options panel into 'media playback' (options governing how media is rendered) and 'media viewer' (options governing the viewer itself like drags and slideshows)
  • added to the new 'media viewer' panel are five checkboxes to turn off the background text in the full media viewer--for the taglist, the top hover, the top-right hover, the notes hover, and the bottom-right index string. if you want, you can have a completely blank background now
  • gave the _help->about_ window a pass. I broke the cluttered first tab into two, and the layout all over is a bit clearer
  • the _help->advanced mode_ option is now available under a new _options->advanced_ tab. this thing covers several dozen things across the program, all insufficiently documented, so the plan is to blow it out into all its granular constituent components on this page!
  • fixed it so an invalid `ApplicationCommand` will still render a string. if you got some jank `ToString()` errors in a shortcuts dialog recently, please try again and let me know what you get. you'll probably want to go into the actual shortcut with the error string and try and see if you can fix what it has set--again, let me know the details please!
  • updated the 'installing and updating' help page to talk clearly about the different versions that have special update instructions, and generally gave the language a pass

  • some url encoding

  • fixed an issue in url encoding-normalisation where urls were not retaining their parameters if their names had certain decoded characters (particularly, this was stuff like the decoded square brackets in `fields[post]=123`). a new unit test will catch this in future
  • url classes and parsers are now careful to encode their example urls any time they are asked for (outside of their respective edit dialogs' "example url(s)" fields, so if you want to work with a human-looking URL in UI, that's fine). this ensures the automatic url-parser linking system works if the parser and url classes have a mish-mash of encoded and non-encoded example URLs. it also fixes some stuff like the multi-column list in the manage url classes dialog when the url class has a decoded example url. this was basically just an ingestion point that I missed in the previous work
  • the edit parser dialog makes sure to properly encode the URL when you do a test pull

  • orphan table tech

  • the _database->db maintenance->clear orphan tables_ command, which could previously only clear out the repository update/processing-tracking tables, can now nuke: the core file list tables in client.db; the core mappings tables in client.mappings.db; the display and storage mappings caches in client.caches.db; the display and storage autocomplete count caches; the ideal and actual tag parent lookup tables in client.caches.db; the ideal and actual tag sibling lookup tables in client.caches.db; and the various tag search tables (except the fts4 stuff) in client.caches.db
  • when this job fires, it now sends orphan tables to the deferred delete system (previously it dropped them immediately, which for a big mappings table is a no-go)

  • boring cleanup

  • cleaned a bunch of db table code for the new orphan table stuff
  • deleted the old 'yaml_dumps' table and all associated methods, which are all now unused
  • added a couple help labels to the "colours" and "style" pages to better explain what is actually going on here

• version 576

  • file access latency

  • the mpv player no longer hangs the UI thread on file load if the file manager is busy. it now just shows a black square until things are freed up. sorry this took so long to fix!
  • the client file storage system has a new two-layer locking mechanism that allows for massively more parallel access, even when files are importing. file imports should lag out file/thumbnail load significantly less
  • the 'check for file orphans' maintenance job is now a significantly less-blocking process. it'll lock each of the 512 subfolders in turn, which will delay some file/thumb access, but it won't need an exclusive write lock on the whole client files manager for the entire job any more
  • also, the 'check for file orphans' job now saves thumbnails, sticking them in a subdirectory of the export location you designate. some users wanted to try using saucenao-type services to try and recover when they had a thumb but no file, so let's see how this works out

  • import options in watchers and gallery downloaders

  • instead of the mysterious 'set options to queries' button, there is now a button beside the 'import options' one that is only visible when the current selection of downloaders has differing file limit or import options than the main page. although this is still a complicated idea, I hope this will make it a little more obvious what is going on
  • I did the same deal for the watchers page, for checker options or the import options
  • it may be that some import options appear to differ after a client restart despite having the same settings. if you get this, let me know the details and I'll fix it!
  • the 'set options to watchers' command now updates note import options
  • fixed gallery imports not always saving changes to their note/tag import options in the main gui session, particularly if they are paused and the client is closed soon after options change
  • improved the import options button's handling of certain options objects when editing, I suspect this fixed some weird edge-case situations of 'I thought I did not set that there' kind of thing, particularly when doing multiple sets of editing to a page and then sub-queries within it
  • the import options button also has a stricter 'set default' command, clearing out old data more thoroughly to help with inter-widget comparisons here

  • misc

  • thanks to a user, we now have support for legacy Microsoft Office documents (.doc, .ppt, .xls), and a framework for other OLE based documents in future
  • this new feature requires the `olefile` library. this is optional, and everyone who runs the normal built release now gets it, but if you run from source you might like to re-run the `setup_venv` script this week so you get it
  • thanks to a user, the danbooru parsers now grab a danbooru post time accurate and precise to the second (previously they were getting 24-hour resolution, I think UTC midnight)
  • uploading large files to the file repository should now use significantly less memory and be far less error prone. due to an in-elegant network request, it was previously timing out the connection if files took too long to upload. the code now streams the upload more cleverly. thanks to the users who helped with this one
  • (tl;dr: if you have a darkmode stylesheet, the colour picker dialog is now fast) it looks like Qt fixed the weird bug that meant certain stylesheets broke the colour picker, so my test that says 'if the user is on Qt 6 and they have a hover-includiig stylesheet, then force a fake stylesheet without that tech before they open the colour-picker dialog and then restore the old one after they close, adding multiple seconds of entry and exit lag to this dialog argh' now no longer applies if you are on Qt 6.6 or later, which is anyone on the built release. let me know if you still have any problems!
  • URLs are now tested against URL Classes by descending order of domain length. this ensures that if you have a URL class for 'api.example.com' and another for 'example.com', and this latter one is set to also apply to subdomains, the specific 'api.example.com' URL Class will be tested first! this was frequently working as desired before, but only for accidental reasons; it is now explicit in all cases

  • boring stuff

  • cleaned up the the regex list in the filename tagging panel, which had some ancient bad code from the wx days that stored the data in the string labels
  • similarly significantly dejanked the 'ListBook' widget used in the options dialog
  • overhauled my four(!!) separate radiobox classes, merging the best of all into one unified class and getting rid of some similar ancient and horrible 'select by label' tech. about twenty or thirty radioboxes across the program, particularly the stuff you see in system predicate panels, now operate on slightly saner principles
  • fixed up the 'default gui session' combobox in the options, which was also inexplicably using ancient tech
  • updated some misc UI typos and unhelpful tooltips
  • refactored some of the client files manager to work with a 'prefix chunk', which will represent an umbrella prefix in the future system that supports overlapping folders and folders with differing prefix lengths'
  • deleted some old client files manager code
  • thanks to a user, the macOS setup_venv is fixed to point at the correct Cocoa/Quartz requirements.txt file

• version 575

  • misc

  • the new 'children' tab now sorts its results by count, and it only shows the top n (default 40) results. you can edit the n under _options->tags_. let me know how this works IRL, as this new count-sorting needs a bit of extra CPU
  • when you ask subscriptions to 'check now', either in the 'edit subscription' or 'edit subscriptions' dialogs, if there is a mix of DEAD and ALIVE subs, it now pops up a quick question dialog asking whether you want to check now for all/alive/dead
  • fixed the (do not) 'alphabetise GET query parameters' URL Class checkbox, which I broke in v569. sorry for the trouble--the new URL encoding handling was accidentally alphabetising all URLs on ingestion. a new unit test will catch this in future, so it shouldn't happen again (issue #1551)
  • thanks to a user, I think we have fixed ICC profile processing when your system ICC Profile is non-sRGB
  • fixed a logical test that was disallowing thumbnail regen on files with no resolution (certain svg, for instance). all un-resolutioned files will now (re)render a thumb to the max bounding thumbnail resolution setting. fingers crossed we'll be able to figure out a ratio solution in future
  • added a _debug->help->gui actions->reload current stylesheet_ menu action. it unloads and reloads the current QSS
  • added a _debug->help->gui actions->reload current gui session_ menu action. it saves the current session and reloads it
  • fixed the rendering of some 16-bit pngs that seem to be getting a slightly different image mode on the new version of PIL
  • the debug 'gui report mode' now reports extensive info about virtual taglist heights. if I have been working with you on taglists, mostly on the manage tags dialog, that spawn without a scrollbar even though they should, please run this mode and then try to capture the error. hit me up and we'll see if the numbers explain what's going on. I may have also simply fixed the bug
  • I think I sped up adding tags to a local tag service that has a lot of siblings/parents
  • updated the default danbooru parsers to get the original and/or translated artist notes. I don't know if a user did this or I did, but my dev machine somehow already had the tech while the defaults did not--if you did this, thanks!
  • added more tweet URL Classes for the default downloader. you should now be able to drag and drop a vxtwitter or fxtwitter URL on the client and it'll work

  • auto-duplicate resolution

  • I have nothing real to show today, but I have a skeleton of code and a good plan on how to get the client resolving easy duplicate pairs by itself. so far, it looks easier than I feared, but, as always, there will be a lot to do. I will keep chipping away at this and will release features in tentative waves for advanced users to play with
  • with this system, I will be launching the very first version of the 'Metadata Conditional' object I have been talking about for a few years. fingers crossed, we'll be able to spam it to all sorts of other places to do 'if the file has x property, then do y' in a standardised way

  • boring stuff

  • refactored the new tag children autocomplete tab to its own class so it can handle its new predicate gubbins and sorted/culled search separately. it is also now aware of the current file location context to give file-domain-sensitive suggestions (falling back to 'all known files' for fast search if things are complicated)
  • fixed a layout issue on file import options panel when a sister page caused it to be taller than it wanted; the help button ended up being the expanding widget jej
  • non-menubar menus and submenus across the program now remove a hanging final separator item, making the logic of forming menu groups a little easier in future
  • the core 'Load image in PIL' method has some better error reporting, and many calls now explicitly tell it a human-readable source description so we can avoid repeats of `DamagedOrUnusualFileException: Could not load the image at "<_io.BytesIO object at 0x000001F60CE45620>"--it was likely malformed!`
  • cleaned up some dict instantiations in `ClientOptions`
  • moved `ClientDuplicates` up to a new `duplicates` module and migrated some duplicate enums over to it from `ClientConstants`
  • removed an old method-wrapper hack that applied the 'load images with PIL' option. I just moved to a global that I set on init and update on options change
  • cleaned some duplicate checking code

• version 574

  • local hashes cache

  • we finally figured out the 'update 404' issue that some PTR-syncing users were getting, where PTR processing would halt with an error about an update file not being available on the server. long story short, SQLite was sometimes crossing a wire in the database on a crash, and this week I add some new maintenance code to fix this and catch it in future
  • the local hash cache has a bunch of new resync/recovery code. it can now efficiently recover from missing hash_ids, excess hash_ids, desynced hash_ids, and even repopulate the master hash table if that guy has missing hash_ids (which can happen after severe db damage due to hard drive failure). it records all recovery info to the log
  • the normal _database->regenerate->local hashes cache_ function now works entirely in this new resync code, making it significantly faster (previously it just deleted and re-added everything). this job also gets a nicer popup with a summary of any problems found
  • when the client recovers from a bad shutdown, it now runs a quick sync on the latest hash_ids added to the local hashes cache to ensure that desync did not occur. fingers crossed, this will work super fast and ensure that we don't get the 404 problem (or related hash_id cross-wire problems) again
  • on repository processing failure and a scheduling of update file maintenance, we now resync the update files in the local hash cache, meaning the 404 problem, if it does happen again, will now fix itself in the normal recovery code
  • on update, everyone is going to get a full local hash cache resync, just to catch any lingering issues here. it should now work super fast!
  • fixed an issue where the local hash and tags caches would not fully reset desynced results on a 'regenerate' call until a client restart

  • misc

  • thanks to a user, the default twitter downloader I added last week now gets full-size images. if you spammed a bunch of URLs last week, I apologise: please do a search for 'imported within the last 7 days/has a twitter url/height=1200px' and then copy/paste the results' tweet URLs into a new urls downloader. because of some special twitter settings, you shouldn't have to set 'download the file even if known url match' in the file import options; the downloader will discover the larger versions and download the full size files with no special settings needed. once done, assuming the file count is the same on both pages, go back to your first page and delete the 1200px tall files. then repeat for width=1200px!
  • the filetype selector in system:filetype now expands to eat extra vertical space if the dialog is resized
  • the filetype selector in file import options is moved a bit and also now expands to eat extra vertical space
  • thanks to a user, the Microsoft document recognition now has fewer false negatives (it was detecting some docs as zips)
  • when setting up an import folder, the dialog will now refuse to OK if you set a path that is 1) above the install dir or db dir or 2) above or below any of your file storage locations. shouldn't be possible to set up an import from your own file storage folder by accident any more
  • added a new 'apply image ICC Profile colour adjustments' checkbox to _options->media_. this simply turns off ICC profile loading and application, for debug purposes

  • boring cleanup

  • the default SQLite page size is now 4096 bytes on Linux and macOS, the SQLite default. it was 1024 previously, but SQLite now recommend 4096 for all platforms. the next time Linux users vacuum any of their databases, they will get fixed. I do not think this is a big deal, so don't rush to force this
  • fixed the last couple dozen missing layout flags across the program, which were ancient artifacts from the wx->Qt conversion
  • fixed the WTFPL licence to be my copyright, lol
  • deleted the local booru service management/UI code
  • deleted the local booru service db/init code
  • deleted the local booru service network code
  • on update, the local booru service will be deleted from the database

• version 573

  • new autocomplete tab, children

  • **this is an experiment. it is jank in form and workflow and may be buggy**
  • the search/edit tag autocomplete dropdowns now have a third tab, 'children', which shows the tag children of the current tag context, whether that is the current search tags or what you are editing
  • the idea is you type 'series:evangelion' but can't remember the character names; now you have a nice list of a bunch of stuff related to what was already entered
  • note you can select this tab real quick just by hitting 'left arrow' on an empty text input
  • this is a first draft, and I would like feedback and ideas, mostly around workflow improvement ideas. it seems to work ok if you have one or two tags with interesting children, but against a big list of stuff, it just becomes another multi-hundred list of spam blah that is difficult to navigate. maybe I could filter it to (and sort by?) the top n most count-heavy results?
  • I wonder if it could also show children on the same level, so if you have 'shinji', it'll also show 'rei' and 'asuka'. I would call this relationship 'siblings', but then we'd be in an even bigger semantic mess
  • also obviously please let me know if this fails anywhere. I think I have it hooked up correct, but some of the code around here is a bit old/messy so some scenario may not update properly
  • don't worry about background lag if you regularly manage lots of tags--it only actually fetches the list of children when you switch to the tab, so you're only spending CPU if you actively engage with it

  • misc

  • a user and I figured out a new twitter tweet downloader using the excellent fxtwitter mirror service. it doesn't do search, but dropping a tweet URL on the client should work again. should handle quoted media and works for multi/mixed-image/video posts, too. note it will nest-pursue quoted tweets, so if there's like fifty in the nested chain, it'll get them all--let me know if this is a big pain and I'll figure out a different solution. I learned that there is another twitter downloader made by a different user on the discord; I have made the update code check for this and not replace it with this if you have it already, and I expect I'll integrate what that can do into these defaults next week
  • the archive/delete and duplicate filters now yes/no confirm when you say to 'forget' at the end of a filtering run
  • the duplicate filter page now only allows you to set the search location to local file domains--so it'll only ever try to search and show pairs for files you actually have
  • fixed the system predicate parsing of `system:duration: has duration` and `system:duration: no duration` when entered by hand, and added a unit test to catch it in future
  • the manage siblings/parents dialogs now have a little shorter minimum height
  • updated some text around the PTR processing in the help--it is only the database proper, the .db files normally in `install_dir/db`, that needs to be on an SSD, and temporary processing slowdowns to 1 row/s are normal
  • touched up some of the 'installing' and 'running from source' help, particularly for some Linux vagaries

  • some build stuff

  • all the builds and the setup_venv scripts are moved from 'python-mpv' to 'mpv', the new name for this library, and the version is updated to 1.0.6, which supports libmpv version >=0.38.x. if you are a windows user and want to live on the edge, feel free to try out this very new libmpv2.dll here, which I have been testing and seems to work well: https://sourceforge.net/projects/mpv-player-windows/files/libmpv/mpv-dev-x86_64-20240421-git-b364e4a.7z/download
  • updated the setup_venv scripts' Qt step to better talk about which Qt version to use for which Python version. it turns out Python 3.12 cannot run something I was recommending for >=3.11, so the whole thing is a lot clearer now

  • boring stuff

  • refactored some question/button dialog stuff
  • fixed up some file domain filtering code in the autocomplete filter and variable names to better specify what is being filtered where

  • local booru deconstruction

  • _reminder: I am removing the local booru, an ancient, mostly undocumented experiment._ if you used it, please check out https://github.com/floogulinc/hyshare for a replacement!
  • the local booru service no longer boots as a server
  • deleted the local booru share cache
  • the local booru review services panel no longer shows nor allows management of its shares
  • deleted the local booru unit tests
  • deleted the local booru help and ancient screenshots

• version 572

  • misc

  • added a new checkbox to _options->files and trash_ to say 'include skipped files when you remove files after archive/delete'
  • thanks to a user, we now have an 'e621' stylsheet in _options->style_. this is the first default stylesheet that uses assets (some checkbox etc.. svgs), which means some users--I think just those who run from source--will need to be careful that their CWD is the hydrus install dir when they boot, or this won't load properly! if you try it and get errors in your log as it tries to load the svgs, let me know!

  • share menu

  • like the 'open' menu a couple weeks ago, the 'share' menu off of thumbnails or the media viewer is rewritten to nicer code. no major differences, but it has a clearer, universal layout, provides more options for 'the currently focused file' vs 'all selected files', is more careful about only providing commands it can deliver on (e.g. no file copy for remote files), and now everything it does is mappable in the shortcut system under the 'media' shortcut set
  • you can now copy a file's thumbnail as a bitmap from this menu!
  • the canvas now supports 'export files'. the 'export files' window just pops on top of it with the one file
  • 'copy file id' is no longer hidden by advanced mode--go nuts!
  • the share menu no longer has 'share on local booru'. the local booru service was an interesting experiment, but I could never find time to properly dev it and there are better answers with the Client API or simple third-party image hosting services that you can drag and drop to. thus, I am finally sunsetting it. I'll strip away its features over the coming weeks until it is completely removed

  • shortcut updates

  • the 'copy file hash' shortcut actions, which used to be four separate things, have been collapsed to one action that has a 'hash type' dropdown (and a 'target' dropdown to select either all selected files or just the currently focused file, which will default to 'all selected' on update, which was the previous behaviour). you can also now set 'pixel_hash' or 'blurhash' as the hash type
  • the 'copy file bitmap' shortcuts have similarly been collapsed down to one action with a dropdown, also with the new 'copy thumbnail' command
  • the 'copy files', 'copy file paths', and 'copy file id' shortcuts now have a dropdown for whether you want all selected files or just the currently focused file. updated commands will default to 'all selected', which was the previous behaviour
  • added a 'copy ipfs multihash' shortcut action, which has this new 'focused vs all selected' parameter and the ipfs service to copy from as its options

  • boring code cleanup

  • wrote a new command for copying arbitrary file hashes, with a new 'file command target'
  • simplified the media hash copying code
  • wrote a new command for copying arbitrary bitmap types
  • combined the bitmap copying code into one shared function call and simplified the surrounding code
  • combined the file and path copying code into shared functions, simplified the code, and added tech for focused vs all selected targeting
  • and the same thing for copying ipfs multihashes
  • wrote a routine to copy a file's thumbnail in the normal clipboard copying pubsub
  • with the recent rounds of simplication, the core thumbnail menu call is now but a mere 600 lines of spaghetti code
  • misc renaming of some enums here so they are more in agreement ('xxx files' instead of 'xxx file', etc...)
  • renamed the various simple commands I have replaced in the past few weeks as 'legacy', so we don't accidentally refer to them again in real code
  • the unit test for 'dateparser decode' is no longer run if dateparser is not in the environment
  • fixed the file metadata parsing unit tests to account for newer ffmpeg, which sees a -10ms different duration on one of the test files, and made the various tests +/-20% lenient to handle this stuff if it comes up again in future

• version 571

  • clean install

  • the recent 'future build' test went well, so I am rolling these updates into the normal release for everyone. on Windows and Linux, the built program is now running Python 3.11, and, on all platforms, updated versions of Qt (UI) and OpenCV (image-processing). there's nothing earth-shattering about these changes, but some things will work better and faster
  • **because of the jump, v570 and v571 have dll conflicts! if you are on Windows or Linux and use the .zip or .tar.zst "Extract" release, you will need to a clean install as here**: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
  • **if you are a Windows installer/macOS App/source user, you do not need to do a clean install, just update as normal**

  • misc

  • when you finish an archive/delete filter and there are several domains you could delete from, the 'commit' buttons are now disabled for 1.2 seconds. this catches you from accidentally spamming enter through a surprise complicated decision
  • under _options->files and trash_, you can now say 'when finishing filtering, always delete from all possible domains', which makes the above decision always single domain. hit this if you do want to spam through this and are fine always deleting from everywhere
  • the client will now, by default, attempt to load truncated images. this was previously off until you set it per-session-on in a debug menu, but is now a checkbox under _options->media_. some weird damaged jpegs and pngs should now load, fingers crossed
  • the 'load images with PIL' setting is now default on for new users and no longer IN TESTING
  • every normal single column text list across the program now copies text better if you explicitly hit ctrl+c/ctrl+insert. they now copy all selected rows (rather than just one), and when the display text differs from the underlying data/sort text, you'll now get the sort text (e.g. on manage urls launched on multiple files, you might see 'site.com/123456 (2)', but now, when it copies, that ' (1)' display cruft is omitted). I spammed this to 22 locations and tested 2 so there are definitely no weird string copy bugs anywhere
  • fixed an issue opening/closing manage parsers, url classes, or url class links if you have url classes with invalid example urls or critically missing default values in your storage
  • the server has a new 'restart_services' command, only triggerable by an admin with service modification ability, which tells all the services on all ports to stop and restart. if there's a new ssl cert, they load the new one

  • client api

  • the 'associate urls' command has a new 'normalise_urls' parameter (default true, which was the behaviour before) to let you force-add un-normalised URLs or URIs or whatever
  • added some unit tests to test this new param
  • client api version is now 64

  • help docs

  • wrote a new help document, 'help my db disappeared.txt' for the db directory that tells you what to do if you boot one day and suddenly get the 'this looks like the first time you ran this program' popup
  • clarified the Windows 'running from source' help a little around 'git' and added a 'here is the Python version you want' link for Win 7 users
  • gave the install help a very light pass, just fixing and updating a few things here and there. I also warn Linux users that the AUR package may throw errors if Arch updates a Qt library or something before we have had a chance to test it (as we have seen a couple times recently), and I generally suggest AUR people run from source manually if they can

• version 570

  • UI stuff

  • wrote a thing to wrap tooltips and applied it everywhere. every tooltip in the program should now wrap to 80 characters
  • the thumbnail view is now better about pausing the current video if you open it externally in various ways
  • the 'open' submenu you get off of a file right-click is now exactly the same for the thumbnail menu and the media viewer menu, with all commands working in either place, the labels are also brushed up a little
  • added a shortcut action for 'open file in web browser' to the media shortcut set
  • added a shortcut action for 'open files in a new duplicates filter page' to the media shortcut set
  • added/updated the shortcut action for 'open similar looking files in a new page' in the media shortcut set. this is now one job that lets you set any distance, and it now works from the media viewer too. all existing `show similar files: 0 (exact)` fixed-distance simple actions will be converted to the new action when you update
  • I removed 'open externally' and 'open in file explorer' shortcuts from the media viewer/preview viewer/thumbnails sets. these sets are technically awkward and were really meant for a different thing, like pause/play or 'close media viewer', and having the media command code duplicated here was getting spammy. if you have any of these now-defunct commands set, please move them up to the general 'media' set, where it'll work everywhere. sorry if this breaks a very complicated set you have, but let's KISS!
  • the 'files' submenu off thumbnails or the media viewer is flattened one level. the 'upload to' remote services stuff still isn't available for the media viewer, but I'll do the same as I did above for that in the near future

  • misc

  • fixed an issue with the 'manage tag siblings/parents' dialogs where the mass-import button was, in 'add or delete' mode, not doing any deletes/rescinds if there were any new pairs in what was being imported. this was probably applying to large regular adds in the UI, also
  • this mass-import button of 'manage tag siblings/parents' also dedupes the pairs coming in. it now shouldn't do anything like 'add, then ask to remove' if you have the same pair twice!
  • the nitter downloaders are removed from the defaults. I can't keep up with whatever the situation is there
  • the style and stylesheet names in the options are now sorted
  • sidecar importers will now work on sidecars that have uppercase .TXT or .JSON extensions

  • more URL stuff (advanced, can be ignored by most)

  • fixed up the recent URL encoding tech to properly follow the encoding exceptions as under RFC 3986. an '@' in an URL shouldn't get messed up now. thanks to the user(s) who helped here
  • incoming URLs can now have a mix of encoded and non-encoded characters and the 'ensure URL is encoded' process will accept it and encode the non-encoded parts, idempotently. it only fails on ingesting a legit decoded percent character that happens to be followed by two hex chars, but that's rare enough we don't really have to worry
  • you can similarly now enter multiple tags in a query text that are a mix of encoded and non-encoded, a mix of %20 and spaces, and it should figure it out
  • the 'ensure URL is encoded' process now applies to GUG-generated URLs, and in the edit GUG UI, you now see the normalised 'for server' URL, with any additional tokens or whatever the URL class has
  • GUGs also try to recognise if their replacement phrase is going into the path or the parameters now, and only force-encodes everything if it looks like our tags are going into a query param
  • ensured that what you paste into an 'edit URL Class' panel's 'example url' section gets encoded before normalisation just as it would in engine
  • the file log right-click now shows both the normalised and request urls under the 'additional urls' section, if they differ from the pretty human URL in the list
  • right-clicking a single item in the downloader search log now previews the specific request URL to be copied

  • boring stuff

  • all instances of URL path or parameter encoding now go through one location that obeys RFC 3986
  • replaced my various uses of the unusual `ParseResult` with `urllib.parse.urlunparse`
  • added a couple unit tests for the improved URL encoding tech
  • added some unit tests for the GUGs' new encoding tech
  • harmonised how a file is opened in the OS file explorer in the media results and media canvas pages. what was previously random hardcode, duplicated internal method calls, and ancient pubsub redirects now all goes thorugh the application command system to a singular isolated media-actioning method
  • did the same harmonisation for opening files externally
  • and for opening files in your web browser, which gets additional new infrastructure so it can plug into the shortcuts system
  • and to a lesser degree the 'open in a new page' and 'open in a new duplicates filter page' commands
  • moved the various gui-side media python files to a new 'gui.media' module. renamed `ClientGUIMedia` to `ClientGUISimpleActions` and `ClientGUIMediaActions` to `ClientGUIModalActions` and shuffled their methods back and forth a bit
  • cleaned up `ClientGUIFunctions` and `ClientGUICommon` and their imports a little with some similar shuffle-refactoring
  • broke up `ClientGUIControls` into a bunch of smaller, defined files, mostly to untangle imports
  • cleaned up how some text and exceptions are split by newlines to handle different sorts of newline, and cleaned up how I fetch the first 'summary' line of text in all cases across the program
  • replaced `os.linesep` with `\n` across the program. Qt only wants `\n` anyway, most logging wants `\n` (and sometimes converts on the fly behind the scenes), and this helps KISS otherwise. I might bring back `os.linesep` for sidecars and stuff if it proves a problem, but most text editors and scripting languages are very happy with `\n`, so we'll see
  • multi-column lists now show multiline tooltips if the underlying text in the cell was originally multiline (although tbh this is rare)

• version 569

  • user contributions

  • thanks to a user, fixed a problem with the recent URL changes that caused downloaders examining multi-file posts to only grab the first file
  • thanks to a user, all the menubar commands that launch a modal dialog are now suffix'd by an ellipsis
  • thanks to a user, fixed an issue regarding KDE 6 quitting the program as soon as the pre-boot 'your database is missing a location, let's find it' repair dialog was ok'd
  • thanks to a user, the application icon is fixed in KDE Plasma Wayland (and anything else that pulls icon from .desktop file). if you have been using a hydrus.desktop file and don't see a program icon, you should rename it to `/usr/share/applications/io.github.hydrusnetwork.hydrus.desktop` . more importantly, if you manage a package for hydrus--please output to this file path instead of `hydrus.desktop` if you make one
  • thanks to a user, updated the `hydrus_client.sh` file to include `"$@"`, which passes parameters given to the .sh file to the .py call

  • more on last week's URL work

  • fixed the 'show the Request URL under "additional urls" submenu' thing on the file log list menu. I screwed up the logic and was effectively testing for when `1 != 1`
  • the converter that generates a Referral URL now operates on the API/redirect conversion principle too--it normalises the Source URL to its 'Request URL' state--keeping defined ephemeral params and filling in defaults but dropping any extra gubbins not asked for--before applying the conversion
  • fixed the 'manage url class' dialog to correctly display an example API/redirect-converted URL based on the new _request url_, not the _normalised url_ (so the api/redirect example will now show the new ephemeral params properly). this was working in requests correctly behind the scenes, it was just the example text box in the dialog that was showing wrong
  • improved the 'is this query text pre-encoded?' test to check for `%hh`, where `h` is a hexadecimal character, instead of the hackier 'is % in it while not followed by whitespace or end of string?'
  • improved/simplified/optimised the overall procedure that figures out if an entered URL is pre-encoded or not. this routine now only runs at the stage where a URL is ingested and it obeys the `%hh` rule. these ingestion points are currently: the text boxes in a urls downloader/simple downloader page; the 'import new sources' function of file log menus; a URL `ContentParser` in the parsing system; the test box in `manage url classes`; and the main gui's 'import url' landing pad, which is used by the drag and drop system, the clipboard watcher, and the client api's 'import url' command. note that this does not occur on 'manage known urls' editing, where you can do what you want with whatever, and I won't coerce it to anything

  • misc

  • fixed a variety of logical cases around >0, =0, !=0, <0 for the `NumberTest` objects I recently applied to system:duration and elsewhere. when it comes to file searching, files that have 'None' duration are now considered equivalent to files that have an explicit 0 duration in all cases. previously, I was trying to thread a needle where '=0' would find null results but <x would not, and it was a mess. now it all works the same way. if you want to search for 'duration < x' and want to exclude still images, either add a filetype pred or slap on 'has duration'
  • improved the stability of the manual file exporter process. it was consulting an object in a thread that it shouldn't have
  • improved the ability of the manual file exporter process to report errors on a very large export that encounters errors after the dialog has closed
  • fixed the 'remember last used default tag service in manage tag dialogs' and its accompanying dropdown not saving their current value on options dialog ok. sorry for the trouble!
  • fixed the system that truncates very long filenames (for export folders and drag and drop exports) on Linux when the exporter is also outputting a sidecar that has a long extra suffix
  • the 'find potential duplicate pairs' routine that runs in idle time now properly obeys the work/rest times in `options->maintenance and processing`. previously, it was just the 'run now' routine that was resting in that way, and the idle thing was just doing a hardcoded 'work for 60 seconds every 10 mins or so'. thanks to the reporting user who cleverly noticed this
  • the `options->connection` page now mentions your proxy needs to be `http://`

  • boring stuff

  • updated the windows setup_venv.bat to allow for custom python or venv locations using parameters. this was so I could set up a multi-python testing situation easier
  • added some unit tests for the new URL encoding gubbins
  • improved un-encoded URL parsing in the downloader when the URL is relative and needs to be joined to the source url
  • improved some URL parsing and ingestion to better handle urls with non-ascii characters in the domain
  • replaced several 'does it start with "http"?' areas with a better and unified scheme/netloc test
  • wrote a routine to split URL paths into path components, and spammed it everywhere so this code is now unified. I expect we'll get a `PathComponent` class at some point, too. there will be a future question about what to do with double slashes, `//` in paths--it turns out the logic has been mixed here, and I think I will probably collapse them to `/` in all cases
  • rewrote an unhealthy call that indirectly caused the above multi-file post parsing problem
  • fixed some None/0 `NumberTest` stuff if you manage to enter '<0' or >-5 and similar
  • I figured out the problems with PyInstaller 6.x and some other stuff, there should be a 'Future Build' alongside this release in github for advanced users to test with

• version 568

  Version 567 was cancelled, its changes folded into 568

  • user contributions

  • thanks to a user, the new docx, pptx, and xlsx support is improved, with better thumbnails (better ratio, better icon itself, and sometimes an actual preview thumbnail for pptx), better file detection (fewer false positives with stuff like ppt templates), and word count for docx and pptx. I am queueing everyone's existing docx and pptx files for a metadata rescan and thumbnail regen on update
  • thanks to a user, the cbz scanner now ignores the `__MACOSX` folder
  • thanks to a user, setting the Qt style in *options->style* should be more reliable (fixing some name case sensitivity issues)
  • thanks to a user, there's a new 'default' dark mode QSS stylesheet that has nicer valid/invalid colours. we'll build on this and try to detect dark mode better in future and auto-switch to this as the base when the application is in dark mode.

  • misc improvements

  • added a 'tag in reverse' checkbox to the new incremental tagger panel. this simply applies the given iterator to the last file first and then works backwards, e.g. 5, 4, 3, 2, 1 for start=1, step=1 on five files
  • all _new_ system:url predicates will have slightly different (standardised) labels, and all these labels should parse correctly in the system predicate parser if you copy/paste
  • the file log's right-click menu, the part where it says 'additional urls', is now more compact and will show the 'request url', if that differs from the main url, either because of the new ephemeral parameters or an api/redirect. it is now much easier to debug the various 'what was actually sent to the server?' problems!
  • you should now be able to enter 'system:has url matching regex (regex with upper case)' and 'system:has url (url with upper case)' and it'll propagate through parsing. this definitely has not™ broken any other predicate parsing. you can enter url class names with upper case if you want, but url class names should now match regardless of letter case
  • if you have added, edited, or deleted any url classes and try to cancel the 'manage url classes' dialog, it will now ask if that is correct
  • added a new EXPERIMENTAL checkbox to _options->tag presentation_ that will replace emojis and other unicode symbol garbage with □. if you have crazy rendering for emoji stuff, try it out
  • the tag summary generators that make thumbnail banners now wash their tags through the 'render tag for user' system, which will apply this new emoji rule and 'replace underscores with spaces'
  • added the 'rating' parser from the default gelbooru 0.2.5 parser to the 0.2.0 parser; this should add for more 'rating' parsing from a variety of boorus

  • misc fixes

  • fixed a typo bug when deleting domain-based timestamps in the edit times dialog
  • fixed the 'system:has url matching class (blah)' predicate edit panel's initialisation. it was always initialising to the top of the list, not remembering the 'default' or 'I want to edit this' value it was initialising with
  • 'manage urls' now asks if it is ok to ok if you have any text still in the input
  • you can now open the 'extra info' button (up top of a media viewer) on a jpeg if that jpeg has no exif or other human-readable metadata (to see just the progressive and subsampling info)
  • updated the QuickSync link to its new home at https://breadthread.duckdns.org/

  • append random text

  • the String Converter has a new step type: 'append random text'. you supply the population (e.g. '0123456789abcdef') and the number of characters (e.g. 16), and it will append 'b2f96e8eda457a1e', and then the next time you check, '1fa591ad9786ea3b', etc... useful if you want to, say, make up a new token

  • URL storage/display changes

  • today I correct a foolish decision I made when I first implemented the hydrus downloader engine--handling and storing URLs internally as 'pretty' decoded text, rather than with the proper ugly '%20" stuff you sometimes see. I now store urls as the 'encoded' variant all the time, and only convert to the pretty version when the user sees it. this improves support for weird URLs and simplifies some behind the scenes. you do not need to do anything, and everything should work pretty much as before, but there is a chance some particularly funky URLs will redownload one more time if your subscription runs into them again (this change breaks some 'known url' checking logic, since what is stored is now slightly different, but this 99% doesn't affect Post URLs, so no big worries)
  • so, while URLs still show pretty in a file/search log, if you copy them to clipboard, you now get the encoded version--pretty much how your web browser address bar works. I have made it show 'pretty' in the file log and search log lists, 'copy url' menu labels, and hyperlink tooltips, but in the more technical 'manage url classes' and 'manage GUGs' and so on where you are actually editing a URL, it shows the encoded version. let me know if I have forgotten to display them pretty anywhere!
  • **IF YOU ARE AN ADVANCED USER WHO MAKES CRAZY URL CLASSES:** since URLs are now stored as the %-encoded version in all cases, component and parameter tests now apply to %-encoding (e.g. you are now testing for `post%5Bid%5D`, not `post[id]`). when your URL Classes update this week, I convert existing path component defaults, parameter names and defaults, and `fixed_text` String Matches for path component names and parameter values to their %-encoded value. I hope this will provide for a clean transition where it matters. unfortunately, if the String Matches were a regex or you were pulling a rabbit out of your hat with edge-case pre-%-encoded default values, I just can't auto-convert that, so please scroll down your crazier URL Classes and see if any say they don't match their example URLs!
  • there's also some GUG work. when you enter a query text like `male/female` or `blonde_hair%20blue_eyes`, some new logic tries to infer whether what you entered is pre-encoded or not. it should handle pretty much everything well unless you have a single-tag query with a legit percent character in the middle (in which case you'll have to enter `%25` instead, but we'll see if it ever happens)
  • these changes simplify the url parsing routine, eliminating plenty of nonsense hackery I've inserted over the years to make things like `6+girls blonde_hair`/`6%2Bgirls+blonde_hair` work with a merged system. this has mostly been a delicate cleanup job; long planned, finally triggered

  • allow all ephemeral parameters

  • URL Classes have a new checkbox, 'keep extra parameters for server', which will determine whether URLs should hang on to undefined parameters in the first stage of normalisation, which governs what is sent to the server. this is now default True on all new URL Classes! existing URL Classes will default True only if the URL Class is a Gallery/Watchable URL without an API/redirect converter (which was essentially the previous hardcoded behaviour). you cannot set this value if the URL has an API/redirect converter

  • allow specific ephemeral parameters

  • alternately, you can now specify single 'ephemeral token' parameters in the new parameter edit dialog. it is just a check box that says 'use this for the request, but don't save it'. these _are_ kept for the API/redirect URL
  • if you are feeling extremely big brain, there is now a String Processor for the default value, if both 'is ephemeral?' is checked and 'default value' is not 'None'. this lets you append/replace your fixed default value with the current time, or, now, just some random hex or something! hence we can now define our own basic one-time token generators for telling caches to give us original quality etc...

  • manage url classes dialog

  • there's a new read-only text field with the 'example url' and 'normalised url' section called 'request url'. this shows either the example URL with its extra, ephemeral parameters, or it will show the API/redirect URL. it shows what will be sent to the server
  • URL Class parameters now have their own edit panel, with everything available in one place, rather than the three-dialogs-in-a-row mess of before. also, the name and value widgets have locked normal/%-encoded text inputs that will live update each other, so you can paste whatever is convenient for you and see a preview either way
  • URL Class path components also have their own edit panel. same deal as for parameters, but a little simpler

  • client api

  • the `/add_urls/get_url_info` command now returns `request_url` value, which is either the 'for server' normalised URL, which may include ephemeral tokens, or the API/redirect URL, just as in the new 'manage url classes' dialog
  • the `/add_files/undelete_files` command now filters the files you give it to make sure that they are actually in your file storage. no more undeleting files you don't have!
  • added a new `/add_files/clear_file_deletion_record` command, which erases deletion records for physically deleted files
  • updated api help docs and unit tests for the above
  • client api version is now 63

  • boring stuff

  • the client is now much more robust if any of its URL Classes do not match their own example URLs. it will boot, to start with (lol), and you can now open the 'manage url classes' dialog without UI error popups. manage url classes now notes which URL Classes do not match their own example URLs, for easy skimming
  • the 'URL Class' class has a new buddy 'Parameter' class to handle param testing
  • simplified some of the guts of URL normalisation, from path/param clipping to how API URL generation is navigated
  • rewrote how the query string of a URL is deconstructed and scanned against your parameters. less chance of edge-case errors/merges and easier to expand in future
  • when you paste a URL, some new normalisation tech tries to figure out if it is pre-encoded or not
  • brushed up the URL Class unit tests to account for the above changes and added new tests for encoding, 'is ephemeral', 'keep extra params for server', default parameter string processors, and simple default parameter values (which must have been missed a long time ago)
  • also broke the monolithic url class unit test into eight smaller (albeit ugly for now) pieces
  • added a unit test for the new 'append random text' converter
  • cleaned up some misc URL Class code

• version 566

  • incremental tagging

  • when you boot a 'manage tags' dialog on multiple files, a new `±` button now lets you do 'incremental tagging'. this is where you, let's say for twenty files, tag them from page:1->page:20. this has been a long time in the works, but now we have thumbnail reorganisation tech, it is now sensible to do.
  • the dialog lets you set a namespace (or none), start point (e.g. you can start tagging at page:19 if you are doing the second chapter etc...), the step (you can count by +2 every file, instead of +1, or even -1 to decrement), the subtag prefix (so you can say 'page:insert-4' or something), and the subtag suffix (for, say, 'page:2 (wip)')
  • the last namespace is remembered between dialog opens, and if the first file in the selection has a number tag in that namespace, that is the number the 'start' will initialise with. a bit of overlap/prep may save time here!
  • the prefix and suffix are remembered between dialog opens
  • a status text gives you a live preview of what you will be adding and says whether any of the files already have exactly those tags or have different tags under the same namespace (which would be possible conflicts, suggesting you are not lined up correct)

  • misc

  • added import support for .docx, .xlsx, and .pptx files (the Microsoft Open XML Formats). they get icons, not much else. they are secretly zips, so **on update, you will be asked if you want to scan your existing zips for these formats**
  • when you move a window to another screen in a maximised state (e.g. on Windows you can do this with win+shift+arrow), the system that remembers window coordinates will now register and save this. the 'restore' window size is preserved from whatever it was on the previous screen while the 'restore' position will try to stay the same on the new monitor (e.g. if it was at (200, 400) on the old monitor, it will try to do the same on the new) as long as the window fits, otherwise it is moved to (20,20) on the new screen
  • the 'edit string converter' panel no longer requires you to enter an example text that can be converted. you can see the error on the dialog, so if you don't want to fix it, or you just need to nip in and out testing things, it is now up to you
  • if the database takes a long time to update, the 'just woke up from sleep' state should no longer trigger. the system thought the long weird early delay was the computer going to sleep
  • the system that gives a popup and then a dialog when you have 165+ (and then 500+ or so) pages open is now removed. this was always a wx thing primarily, and Qt is much happier about having a whole load of UI elements. the main problem here is now memory blot and UI-update lag. this is now in the user's hands alone, no more bothering from me (unless it becomes a new problem, and I'll figure out a better warning test/system)

  • boring code cleanup

  • neatened how some manage tags ui is initialised. there's a hair of a chance this fixes the 'the manage tags dialog taglist is cut off sometimes' bug
  • neatened how some pending content updates are held in manage tags
  • manage tags dialogs now receive their media list in the same order as the underlying thumbnail selection, ha ha ha
  • untangled some of the presentation import options. stuff like 'is new or in inbox' gets slightly better description labels and cleaner actual logic code
  • fixed some type issues, some typo'd pubsubs, and other misc linting
  • tried last week's aborted github build update again. the build is now Node 20 compatible

• version 565

  • tag sorting bonanza

  • _options->sort/collect_ now offers four places to customise default tag sort. instead of having one default sort for everything, there's now sort for search pages, media viewers, and the manage tags dialogs launched off of them

  • tag filter

  • when you copy namespaces from the tag filter list, it now copies the actual underlying data text like `character:`, which you can paste in elsewhere, rather than the pretty `"character" tags` display text
  • brushed up some of the UI and help text on the tag filter UI
  • fixed a couple places where the tag copy menus were trying to let you copy an empty string, which ended up with `-invalid label-`
  • fixed some extremely janked-out logic in the tag filter that was sending `(un)namespaced tags` to the 'except for these' advanced whitelist in many cases. it was technically ok, but not ideal and overall inhuman

  • concatenated source urls

  • on rule34.xxx and probably some other places, when the file has multiple source urls, the gelbooru-style parsers were pulling the urls in the format [ A, B, C, 'A B C' ], adding this weird extra string concatenation that is obviously invalid. I fixed the parsers so it won't happen again
  • **on update, you are going to get a couple of yes/no dialogs asking if you want to scan for and delete existing instances of these URLs**. if you have a big client, it will take some time to do this scan. the yes/no dialogs will auto-yes after ten minutes, so if you are doing a headless update via docker or something, please be patient--it will go through

  • note sidecars

  • the note->media sidecar exporter module now has a 'forced name' input. if you want to parse a single note from a .txt or .json that doesn't have a name, you can now force it
  • the sidecard txt separator dropdown in the .txt importer module now has a 'four pipes (||||)' entry in the dropdown as a quick-select beside 'newline'. four pipes is a useful separator of multi-line notes content since it almost certainly won't come up in a normal note
  • some tooltips and stuff are updated around here to better explain what the hell is going on
  • added a unit test to test the forced name

  • misc

  • to help the recent shortcuts change that merged `numpad` variants of + and left arrow and so on into being seen as the `unmodified` variants, if you have a saved shortcut that _is_ still the `numpad` variant, it will now match the `unmodified` input when the merge mode is on. just means you don't have to remap everything with this mode on--everything merged matches everything
  • added 'copy file known urls' to the 'media' shortcut set
  • I forgot to mention last week that we figured out more native global menubar tech (where the top menubar of the program will embed into your OS's top system menubar) in last week's release, for non-macOS (some versions of Linux) users. the new checkbox is under _gui->Use Native MenuBar_. it defaults to on for macOS and off for everyone else, but feel free to try it. there was a related 'my menubar is now messed up, why?' bug that hit some people in v564 that is fixed today. sorry if you got boshed by this, since it was tricky to manually fix. in future, note you can hit ctrl+p in a default client to bring up the command palette, and then you can type 'options' and can open the options that way, if your menubar isn't working!
  • fixed the `ideal usage` calculation in _database->move media files_ when there are three or more competing storage locations with two or more having a max size that is exceeded by their weight, and one or more having a max size that is only exceeded by their weight a little bit. due to a mistake in how total remaining weight was calculated in the little behind the scenes elimination game here, a location in this situation was exceeding its max size amount by a multiple of `1/(1-total_normalised_weight_of_restricted_locations)`, typically +10-30%. thank you for the report here, it was interesting to figure out!
  • I removed a hack that made the repositories (like the PTR) work for users running super old versions of the client. the hack has now been in place for more than a year. if you run into repository syncing problems, please update to after v511!
  • fixed a dumb status line in the 'check for missing/invalid files' checker thah was double-counting bad files in the popup
  • fixed some media duration 'second' components being rendered with extraneous .0, like '30.0 seconds'
  • fixed right-clicking on some empty system predicates like 'system:duration' (those I overhauled the other week with NumberTests)
  • fixed a db routine that fetches a huge table in pieces to not repeat a few rows when the ids it is fetching are non-contiguous, and to report the correct quantity of work done as a result (it was saying like 17,563/17,562)
  • the new _help->about_ Qt platformName addition will now say if the actual platformName differs from the running platformName (e.g. if it was set otherwise with a Qt launch parameter)

  • client api

  • just a small thing, but the under-documented `/manage_database/get_client_options` call now says the four types of default tag sort. I left the old key, `default_tag_sort`, in so as not to break stuff, but it is just a copy of the `search_page` variant in the new `default_tag_sort_xxx` foursome
  • client api version is now 62

• version 564

  • more macOS work

  • thanks to a user, we have more macOS features:
  • macOS users get a new shortcut action, default Space, that uses Quick Look to preview a thumbnail like you can in Finder. **all existing users will get the new shortcut!**
  • the hydrus .app now has the version number in Get Info
  • **macOS users who run from source should rebuild their venvs this week!** if you don't, then trying this new Quick Look feature will just give you an error notification

  • new fuzzy operator math in system predicates

  • the system predicates for width, height, num_notes, num_words, num_urls, num_frames, duration, and framerate now support two different kinds of approximate equals, ≈: absolute (±x), and percentage (±x%). previously, the ≈ secretly just did ±15% in all cases (issue #1468)
  • all `system:framerate=x` searches are now converted to `±5%`, which is what they were behind the scenes. `!=` framerate stuff is no longer supported, so if you happened to use it, it is now converted to `<` just as a valid fallback
  • `system:duration` gets the same thing, `±5%`. it wasn't doing this behind the scenes before, but it should have been!
  • `system:duration` also now allows hours and minutes input, if you need longer!
  • for now, the parsing system is not updated to specify the % or absolute ± values. it will remain the same as the old system, with ±15% as the default for a `~=` input
  • there's still a little borked logic in these combined types. if you search `< 3 URLs`, that will return files with 0 URLs, and same for `num_notes`, but if you search `< 200px width` or any of the others I changed this week, that won't return a PDF that has no width (although it will return a damaged file that reports 0 width specifically). I am going to think about this, since there isn't an easy one-size-fits-all-solution to marry what is technically correct with what is actually convenient. I'll probably add a checkbox that says whether to include 'Null' values or not and default that True/False depending on the situation; let me know what you think!

  • misc

  • I have taken out Space as the default for archive/delete filter 'keep' and duplicate filter 'this is better, delete other'. Space is now exclusively, by default, media pause/play. **I am going to set this to existing users too, deleting/overwriting what Space does for you, if you are still set to the defaults**
  • integer percentages are now rendered without the trailing `.0`. `15%`, not `15.0%`
  • when you 'open externally', 'open in web browser', or 'open path' from a thumbnail, the preview viewer now pauses rather than clears completely
  • fixed the edit shortcut panel ALWAYS showing the new (home/end/left/right/to focus) dropdown for thumbnail dropdown, arrgh
  • I fixed a stupid typo that was breaking file repository file deletes
  • `help->about` now shows the Qt platformName
  • added a note about bad Wayland support to the Linux 'installing' help document
  • the guy who wrote the `Fixing_Hydrus_Random_Crashes_Under_Linux` document has updated it with new information, particularly related to running hydrus fast using virtual memory on small, underpowered computers

  • client api

  • thanks to a user, the undocumented API call that returns info on importer pages now includes the sha256 file hash in each import object Object
  • although it is a tiny change, let's nonetheless update the Client API version to 61

  • boring predicate overhaul work

  • updated the `NumberTest` object to hold specific percentage and absolute ± values
  • updated the `NumberTest` object to render itself to any number format, for instance pixels vs kilobytes vs a time delta
  • updated the `Predicate` object for system preds width, height, num_notes, num_words, num_urls, num_frames, duration, and framerate to store their operator and value as a `NumberTest`, and updated predicate string rendering, parsing, editing, database-level predicate handling
  • wrote new widgets to edit `NumberTest`s of various sorts and spammed them to these (operator, value) system predicate UI panels. we are finally clearing out some 8+-year-old jank here
  • rewrote the `num_notes` database search logic to use `NumberTest`s
  • the system preds for height, width, and framerate now say 'has x' and 'no x' when set to `>0` or `=0`, although what these really mean is not perfectly defined

• version 563

  • macOS improvements

  • Thanks to a user, we have multiple improvements for macOS!
  • There is a new icon for the macOS .app build of hydrus
  • The macOS app will now appear as "Hydrus" in the menu bar instead of "Hydrus Network"
  • - Use the native global menu bar on macOS and some Linux desktop environments
  • - "options" will now appear as "Preferences..." and be under the Hydrus menu on macOS
  • - "exit" will now appear as "Quit Hydrus" and be under the Hydrus menu on macOS
  • "exit and force shutdown maintenance", "restart", and "shortcuts" will now be under the Hydrus menu on macOS
  • The hydrus system tray icon is now enabled for macOS and "minimise to system tray" will be in the Hydrus menu when in advanced mode
  • macOS debug dialog menus are now disabled by default
  • The macOS build of hydrus now uses pyoxidizer 0.24.0 and Python 3.10
  • The command palette and hyperlinks colors in the default Qt stylesheet now use palette based colors that should change based on the Qt style
  • one thing hydev did: on macOS, Cmd+W _should_ now close any dialog or non-main-gui window, just like the Escape key

  • shortcuts

  • by default, Alt+Home/End/Left/Right now does the new thumbnail rearranging. **assuming they do not conflict with an existing mapping, all users will recieve this on update**
  • by default, the shortcuts system now converts all non-number 'numpad' inputs (e.g. 'numpad Home', 'numpad Return', 'numpad Left') to just be normal inputs. a bunch of different keyboards have whack numpad assignments for non-numpad keys, so if it isn't a number, let's not, by default, make a fuss over the distinction. you can return to the old behaviour by unchecking the new checkbox under _file->shortcuts_
  • the default shortcuts now no longer spam numpad variants anywhere. existing users can delete the surplus mappings (under 'thumbnails' and maybe some of the 'media' sets) if they like

  • some UI QoL

  • the _tag service_ menu button that appears in the autocomplete panel and sometimes some other places in advanced mode now shows a proper check mark in its menu beside its current value
  • the _location context_ menu button on the other side of an autocomplete panel and some other places also now shows a check mark in its menu beside its current value
  • the `OR` button on search autocomplete that creates new OR predicates now inherits the current file search domain. it was previously defaulting at all times to the fallback file domain and 'all known tags'
  • the current search predicates list also now inherits the file search domain when you edit an OR predicate currently in use, same deal
  • removed the 'favourites' submenu from the taglist menu when no tags are selected
  • in any import context, the file log's arrow menu now supports deleting all the 'unknown' (outstanding, unstarted) items or setting them all to 'skipped'. the 'abort imports' button (with the stop icon) in HDD and urls import pages is removed

  • misc

  • fixed yet another dumb problem with the datetime control's paste button--although the paste was now 'working' on the UI side, the control wasn't saving that result on dialog ok. the fixes both the datetime button and the modified/file service time multi-column list editing
  • a core asynchronous thread-checking timer in the program has been rewritten from a 20ms-resolution busy-wait to a <1ms proper wait/notify system. a bunch of stuff that works in a thread is now much faster to recognise that blocking UI work is done, and it is more thread-polite about how it does it!
  • in the `setup_venv` scripts, if it needs to delete an old venv directory but fails to do so, the script now dumps out with an error saying 'hey, you probably have it open in a terminal/IDE, please close that and try again'. previously, it would just charge on and produce an odd file permission error as, e.g., the new venv setup tried to overwrite the in-use python exe
  • added a `help->debug->gui->isolate existing mpv widgets` command to force regeneration of mpv windows and help test-out/hack-fix various 'every other of my mpv views has no audio' and 'my mpv loses xxx property after a system sleep/wake cycle' problems. if I've been working with you on this stuff, please give it a go and let me know if new mpv window creation is good or what!
  • added a `BUGFIX: Disable off-screen window rescue` checkbox to `options->gui` that stops windows that think they are spawning off-screen from repositioning to a known safe screen. several Qt versions have had trouble with enumerating all the screens in a multiple monitor setup and thus the safe coordinate space, so if you have been hit by false positives here, you can now turn it off! (issue #1511)
  • fixed another couple instances of error texts with empty formatting braces `{}`

  • tag repository

  • mapping petitions fetched from the server will now max out at 500k mapping rows or 10k unique tags or ten seconds of construction time. we had a 250k-unique-tag petition this last week and it broke something, so I'm slapping a bunch of safety rails on. let me know if these are too strict, too liberal, or if it messes with the fetch workflow at all--I don't _think_ it will, but we'll see

  • build stuff

  • now they have had time to breathe, I optimised the recently split Github build scripts. the 'send to an ubuntu runner and then upload' step is now removed from all three, so they are natively uploaded in the first runner step. it works just a little nicer and faster now, although it did require learning how to truncate and export a variable to the Github Environment Variables file in Powershell, aiiieeeee
  • also, Github is moving from Node 16 to Node 20 soon, and I have moved two of the four actions we rely on to their newer v20 versions. a third action should be ready to update next week, and another, a general download file function, I have replaced with curl (for macOS) and Powershell's magical Invoke-WebRequest adventure

• version 562

  • misc

  • page tab drag and drops will now not start unless the click has lasted more than 100ms
  • same for thumbnail drag and drop--it perviously did a 20 pixel deadzone, but time checks detect accidental/spastic clicks better and stops false negatives when you start dragging on certain edges
  • added a 'BUGFIX: disable page tab drag and drop' setting to _options->gui pages_. while adding this, I may have accidentally fixed the issue I wanted to investigate (rare hangs on page DnD)
  • the manage tags dialog now shows the current count of tags for each page tab, and, if there are outstanding changes, shows an asterisk
  • the `migrate database` dialog is renamed `move media files`

  • fixes

  • fixed the basic copy/paste in the single 'edit datetime' panel, wich was often raising a dumb error. this thing also now exports millisecond data (issue #1520)
  • I am pretty sure I fixed the column-resizing problem in the very new PySide6 (Qt) 6.6.1, which it seems AUR users were recently updated to in an automatic OS update. all columns were setting to 100px width on initialisation. I think it is now safe to try out 6.6.1. I am still not sure why it was doing this, but some extra safeguards seem to have fixed it and also not broken things for <=6.6.0, so let me know what you run into! if you were affected by this, recall that you can right-click on any multi-column list header and say 'reset widths' to get something sensible back here
  • when exporting files, the max size is now clipped another 84 characters (64 + 20 more, which usually ends up about 150 characters max for the output filename), in order to give padding for longer sidecar suffixes and also avoid going right to the filesystem limit, which broadly isn't sensible
  • I think I fixed an issue where the mouse could stay hidden, perhaps, just on Wayland, after closing the media viewer with your keyboard (issue #1518)
  • fixed inc/dec ratings in the media viewer not updating their tooltips on new media correctly
  • if you hit 'open this location' on the export files window and the location does not exist, you now get a nice messagebox rather than a semi-silent error

  • analyze

  • background: some databases that process the PTR superfast or otherwise import a lot of data to a new file domain sometimes encounter massively massively slow tag update actions (typically tag-delete when the tags involved have siblings/parents), so I want to make the critical 'ANALYZE' call more timely
  • the 'analyze' database maintenance call will be soft-called far more regularly during normal repository processing, not just on first sync
  • sped up how some pre-analyze calculation is done
  • the size limit for automatic database analyze maintenance is raised from 100k rows to 10M
  • I hope to do more work here in future, probably making a review panel like we did for vacuum
  • if your repository processing sometimes hangs your whole damn client for 10-15 minutes, hit _database->db maintenance->analyze->full_! this job may take 30-60 minutes to finish

  • boring code cleanup

  • finished the HG->CG.client_controller refactor I started last week. this was a thousand lines changed from one braindead format to another, but it will be a useful step towards untangling the hell-nest import hierarchy
  • did a scattering of the clientinterface typing, getting a feel for where I want to take this
  • deleted the old in-client server-test's 'boot' variant; this is no longer used and was always super hacky to maintain
  • I removed an old basic error raising routine that would sometimes kick in when a hash definition is missing. this routine now always fills in the missing data with garbage and does its best to recover the invalid situation automatically, with decent logging, while still informing the user that things are well busted m8. it isn't the user's job to fix this, and there is no good fix anyway, so no point halting work and giving it to the user to figure out!

• version 561

  • rearranging thumbnails

  • on the thumbnail menu, there is a new 'move' submenu. you can move the current selection of files to the start or end of the media list, or to one before or after the earliest selected file, or to the file you right-clicked on to create the menu, or to the first file's position if the selection is not contiguous. if the selection is non-contiguous, it will be made so in the move
  • added these rearrange commands to the shortcuts system, as 'move thumbnails' under the 'thumbnails' set. I wasn't sure whether to add some default shortcuts, like ctrl+numpad 7/3/4/6 for home/end/left/right or something--let me know what you think

  • misc

  • thanks to user help, fixed a stupid typo from last week that caused some bad errors (including crashes, in some cases) when doing non-simple duplicate filtering (issue #1514). this is the issue the v560a hotfix was made for
  • fixed another stupid content update typo that was causing 'already in db' results to not get metadata updates
  • as a hardcoded shortcut, Ctrl+C or Ctrl+Insert now copies the currently selected tags in any taglist. it'll output the full tag/predicate text, with namespace, no counts
  • I've shortened some thumbnail/media-viewer menu labels, made the 'delete' line into a submenu, and ensured the top info line is always a short variant, with detailed info bumped off to the submenu off the top line. I hate how these menus are often super-wide and thus a pain to navigate to the submenus, so let me know what situations still make them wide
  • the file log arrow button menu now has entries for 'delete already in db' and 'delete everything'
  • the 'add these tags to the favourites list?' yes/no now only fires if you try to add more than five tags ot once
  • the various dialogs in the client that auto-yes or auto-no now show a live countdown in their title string
  • the window position saving system is now stricter about what it records. maximised and fullscreen state is only saved if 'remember size' is false, and the last size/position is not saved at all if 'remember size/position' is false (previously, it would save these values but not restore them, but let's try being more precise here)
  • fixed a 'omg what happened, closing the window now' error in the duplicate filter if you try to 'go back' while it is loading a new set of pairs to show
  • fixed the 'vacuum db' command to correctly save 'last vacuumed time' for all files vacuumed in a job, not just the last!
  • whenever a `copy2` file copy (which includes copying file times and permission bits) fails for permission reasons, hydrus now falls back to a normal `copy` and logs the failure, including the modified time that failed to copy (which is the bit we actually care about here)

  • db update stuff

  • if there is a known bitrot issue on update, you now get a nicer error message. rather than the actual error, you are now told which version is safe to update to. to christen this system, I've added a check for the recent millisecond timestamp conversion, which caused some issues for users updating older clients. **if your client is v551 or older and you try to update to v561 or later, you will be told to update to v558 first.** sorry for the inconvenience here, and thank you for the reports (issue #1512)
  • if you try to boot a database more than 50 versions earlier than the code, the client-based version popups now happen in the correct order, with the >50 exception firing before the >15 warning
  • when an update asks a not-super-important yes/no question, I will now make it auto-yes or auto-no after ten minutes with the recommended value. this will ensure that automatic updaters will still progress (previously, they were hanging forever!)

  • some downloader stuff

  • thanks to a user, the derpibooru now fetches the post description as a note and the source as an associable URL. I tweaked the submitted stuff a bit, simplifying the parsing and discluding 'No description provided.' notes
  • thanks to a user, the e621 parser can now grab files from posts where the (spicy, I think) content is normally not shown due to a guest login. the posts still won't show up in guest-login gallery searches, so this won't alter your normal results, but if you run into a post like this in your browser and drag-and-drop it onto the client, it now works
  • I tried to improve the parsing system's de-newlining. this thing is a long-time hack--I've never liked it and I want to replace it with proper multi-line support--but for now I've made sure the de-newliner strips each line of leading/trailing whitespace and discards empty lines. the mode that _doesn't_ collapse newlines (note parsing, for the most part) now _does_ strip leading/trailing newlines along with other whitespace, meaning you no longer have to try and strip extra `

    ` and `
    ` tags and stuff yourself when grabbing notes. also, the formula UI where it says 'Newlines are collapsed before...' now says when it won't be collapsing newlines due to it being a note parser

  • the String Match processing step now explicitly removes newlines before it runs, meaning it can still catch multi-line notes properly. you can now run a proper regex on a multi-line note

  • boring cleanup

  • optimised some thumbnail handling code, stuff like fetching the current list of sorted selected media
  • large collections will be a little faster to select and otherwise do operations on
  • sketched out a new `ClientGlobals` and client controller interface and started refactoring various HG.client_controller to the new CG. this makes no important running changes, but it cleans the messy HG file and will help future coding and type checking in the IDE as it is fleshed out
  • added some help text to the edit file maintenance panel and fixed some gonk layout in the 'add new work' panel
  • fixed some instances of the 'unknown' import status showing as a blank string
  • fixed an error message in the export folder export job that fired when a file to be exported is missing--it was just giving blank instead of the file hash, and its direction to file maintenance was old and unclear

• version 560

  • editing times for multiple files

  • the 'edit times' dialog is now available when you select multiple files. it will show and apply time data for all of those files at once. when the files have different times, the various widgets and panels will show ranges and a count of how many files do and don't have that particular time type
  • when you open the edit times dialog on more than one file, every time control now has a 'cascade step' section, where you can set a time delta, e.g. 100 milliseconds, and then, on dialog ok, each file in the selection that launched the dialog will be set that much successively later than the previous, obviously in the order they are currently in. this is a way of forcing/normalising file sorts based on time. negative values are allowed!
  • when the edit times dialog is set to change more than 100 total times, it now verifies with the user that this is correct on dialog ok
  • when the edit times dialog sets a lot of modified dates to files (i.e. actually writing them to your file system), this now happens in a non-gui thread and now makes a cancellable progress popup after a few seconds

  • misc

  • fixed the 'imported to' timestamp for files migrated to other local file domains, which were one of the ones incorrectly set, as expected, to 54 years ago. in the database update, I also fix all the wrongly saved ones from v559
  • mr bones and the file history chart are now under the 'database' menu
  • fixed an issue with the file history chart not maintaining the `show_deleted = False` state through search refreshes
  • there's a new checkbox under `files and trash`, `Remove files from view when they are moved to another local file domain`. this re-introduces the unintended behaviour that I fixed recently when 'remove when trashed' was set, but now targeted specifically for that situation. if you use multiple local file domans a bunch and want files to disappear when you shoot them to a place you aren't looking at, give it a go and let me know how it works for you
  • fixed a regression from my 'remove when trashed' fix where deleting collections with this option on would leave crazy ghost thumbnails behind. collections that are completely emptied should now properly remove themselves in all content update situations
  • the gallery downloader page 'cog' icon now has a 'do not allow new duplicates' option, which will discard any (query_text,source) pairs you try to enter if they already exist in the list. this option is remembered through restarts
  • added 'sort by pixel hash' to the file sort menu. it isn't super helpful, but it'll show pairs of exact-matching files next to each other amongst a sea of noise. I may expose perceptual hashes in a similar way in future, which would be more useful, but thumbnails don't have their phashes quickly available atm, so maybe only when there are other reasons to add that overhead
  • fixed the `setup_venv.sh` and `setup_venv.command` files' custom qtpy and PySide6 (Qt stuff) version installer! there was a dumb typo, sorry for the trouble
  • thanks to a user, the derpibooru parser now grabs `fanfic`, `spoiler`, and `error` tags

  • boring cleanup

  • neatened up how non-thumbnail-generatable files (e.g. rtf) present their default thumbs and refactored the code a little
  • when a file's thumbnail is unavailable but the filetype is known (e.g. you are looking at records of deleted files that have no blurhash), hydrus should now deliver that file's default thumb instead
  • unified this thumbnail-defaulting code a little more, fixing fetching for some weirder files and deduplicating some messy areas. the client thumbnail cache should be better about delivering the right unusual thumbnail now and as future filetypes are added
  • added an 'image.png' to serve as a nicer fallback for various thumbnail-undeliverable but known-image files
  • fixed rtf files not providing their rtf thumbnail in the Client API
  • fixed up some ancient local booru thumbnail fetching code
  • cleaned up some messy dialog launches that were having to navigate single/collected media in an awkward way
  • removed the TestFunctions unit test stub, which was of diminishing use

  • boring cleanup, time code

  • updated the DateTime control and button to handle multiple times at once, and updated the edit timestamps dialog itself similarly throughout (this took a day and a half lol)
  • rejiggered the DateTime widgets to handle a nice new object to hold the multiple times' range, since it was all getting messy
  • rejiggered the time content update pipeline from top to bottom to take multiple hashes per content update, so applying the same timestamp to a thousand files should still be pretty quick
  • fixed up various timestamp_ms->QtDateTime conversions so they all include local timezone info. also fixed the datetime widget so it returns properly local-timezone'd datetimes. I can no longer easily reproduce a particular time that jumps an hour every time you open it (due to retroactive summer-time fun)
  • harmonised some older datestring conversions to come out 2023-06-30 instead of 2023/06/30
  • fixed some time string calculations to handle our new sub-second times better
  • updated the time delta widget to handle negative numbers

  • boring cleanup, content updates

  • moved all `ContentUpdate` gubbins out of the hydrus module scope; it is now client only
  • made a new `ClientContentUpdates.py` to collect all content update code and refactored stuff there
  • wrote a new `ContentUpdatePackage` to replace the ancient `service_keys_to_content_updates` structure. various hacky or ad-hoc processing and presentation is now gathered under this new object, and I refactor-spammed it across the program, with too many individual changes to talk about in detail

  • client api

  • the new `set_time` call has some additional safety rails. you can add (or delete) 'web domain' timestamps any time, but you now cannot add or delete any of the others, only edit when they already exist
  • updated the client api unit tests and help to account for this
  • the client api is now version 60

• version 559

  • millisecond timestamps

  • since the program started, the database and code has generally handled timestamps as an integer (i.e. whole number, no fractions) count of the number of seconds since 1970. this is a very common system, but one drawback is it cannot track any amount of time less than a second. when a very fast import in hydrus imports two files in the same second, they then get the exact same import time and thus when you sort by import time, the two files don't know which should be truly first and they may sort either way. this week I have moved the database to store all file timestamps (archived time, imported time, etc...) with millisecond resolution. you do not have to do anything, and very little actually changes frontend, but your update may take a minute or two
  • whenever you sort by 'import time' now, we shouldn't get anymore switcheroos
  • the 'manage times' dialog now has millisecond display and edit widgets to reflect this, but in most places across the client, you'll see the same time labels as before
  • I changed a **ton** of code this week. all simple changes, but I'm sure a typo has slipped through somewhere. if you see a file with a 'last viewed time' of '54 years ago', let me know!

  • time details

  • this section is just a big list so I have somewhat of a record of what I did. you can broadly ignore it
  • updated `vacuum_timestamps` to `timestamp_ms` and adjusted read/write and the dialog handling to ms
  • updated `analyse_timestamps` to `timestamp_ms` and adjusted read/write to ms
  • updated `json_dumps_named` to `timestamp_ms` and adjusted read/write and some UI-level gubbins around session loading and saving to ms
  • updated `recent_tags` to `timestamp_ms` and adjusted the whole system to ms
  • updated `file_viewing_stats` to `last_viewed_timestamp_ms` and adjusted read/write to ms
  • updated `file_modified_timestamps` to `file_modified_timestamp_ms` and adjusted read/write to ms, including to and from the disk
  • updated `file_domain_modified_timestamps` to `file_modified_timestamp_ms` and adjusted read/write to ms
  • updated `archive_timestamps` to `archived_timestamp_ms` and adjusted read/write to ms
  • updated all the current- and deleted-file tables for all file services to use ms (`timestamp_ms`, `timestamp_ms`, and `original_timestamp_ms`) and adjusted _all_ database file storage, search, and update to work in ms
  • updated the `ClientDBFilesTimestamps` db module to use ms timestamps throughout
  • updated the `ClientDBFilesViewingStats` db module to use ms timestamps throughout
  • updated the `ClientDBFilesStorage` db module to use ms timestamps throughout
  • updated the controller timestamp tracker and all callers to use ms timestamps throughout
  • renamed `TimestampsManager` to `TimesManager` and `times_manager` across the program
  • updated the `TimesManager` and all of its calls and callers in general to work in ms. too much stuff to list here
  • the `TimestampData` object is now converted to ms, and since it does other jobs than a raw number, the various calls it is involved in are generally renamed from 'timestamp' to 'time'
  • the file viewing stats manager now tracks 'last viewed time' as ms, and the update pipeline is also updated
  • the locations manager now handles all file times in ms, and all the archive/add/delete pipelines are also updated
  • wrote some MS-based variants of the core time functions for spamming around here, including for both Qt `QDateTime` and python `datetime`
  • updated the main datetime edit panel, button, and widget to handle millisecond display and editing
  • fleshed out a ton of ambiguous variable names to the new strict time/timestamp/timestamp_ms system
  • wrote a clean transition method between ms<->s that accounts for various None situations and spammed it everywhere
  • fixed up some ill-advised timestamp data juggling in the time edit UI

  • what still has second-resolution

  • the parsing system (and hence downloaded files' source times)
  • the sidecar system's time stuff, both import and export
  • file viewtime durations
  • the server and the hydrus network protocol in general
  • Mr. Bones and the File History chart
  • almost all the actual UI labels. I'm not going to spam milliseconds at you outside of the time edit UI
  • almost all the general maintenance timers, sleepers, and grunt-work code across the program

  • client api

  • the `file_metadata` call has a new parameter, `include_milliseconds`, which turns the integer `1704419632` timestamps into floats with three sig figs `1704419632.154`, representing all the changes this week
  • a new permission, `edit file times` is added, with value `11`
  • a new command, `/edit_times/set_time` now lets you set any of the file times you see in the _manage times_ dialog. you can send it second- or millisecond-based timestamps
  • the client api help is updated for all this, particularly the new section here https://hydrusnetwork.github.io/hydrus/developer_api.html#edit_times_set_time
  • added unit tests for this
  • the client api version is now 59

  • misc

  • the sankaku parsers, GUGs, and custom header/bandwidth rules are removed from the defaults, so new users will not see them. none of this stuff works well/at all any more, especially in recent weeks. for sites that are so difficult to download from, if there isn't a nice solution on the shared downloader repo, https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts, I recommend going with a more robust solution like gallery-dl or just finding the content elsewhere
  • when there are multiple 'system:known url' predicates in a search, I now ensure the faster types run first, reducing the search domain for the slower, later ones. if you have a 'regex' 'known url' predicate, try tossing in a matching 'domain' one--it should accelerate it hugely, every time
  • fixed a bug in the autocomplete dropdown where it was not removing no-longer-valid file services from the location button after their deletion from _manage services_ until program restart (which was causing some harmless but unwelcome database errors). it should now remove them instantly, and may even end up on the rare 'nothing' domain
  • the duplicate filter will no longer mention pixel-perfect pngs being a waste of space against static gifs--this isn't necessarily true
  • the default height of the 'read' autocomplete result list is now 21 rows, so `system:time` and `system:urls` are no longer subtly obscured by default. for existing users, that's under _options->search_
  • in the 'running from source' requirements.txts, I bumped the 'new' and 'test' versions for python-mpv to 1.0.4/1.0.5. the newest python-mpv does not need you to rename libmpv-2.dll to mpv-2.dll, which will be one less annoying thing to do in future. I've also been testing this extremely new dll this week and ran into no problems, if you are also a Windows source user and would like to try it too: https://sourceforge.net/projects/mpv-player-windows/files/libmpv/mpv-dev-x86_64-20231231-git-abc2a74.7z . I also tried out Qt 6.6.1, but I just discovered a column-sizing bug I want to sort out before I roll it out to the wider community
  • updated the sqlite dll that gets bundled into the windows release to 3.44.2. the sqlite3.exe is updated too

• version 558

  • user contributions

  • thanks to a user, we now have rtf support! no word count yet, but it should be doable in future.
  • thanks to a user, ctrl+p and ctrl+n now move the tag listbox selection up and down, in case the arrow keys aren't what you want. it also works on the tag autocomplete results from the text input
  • added a link to 'Hydra Vista', https://github.com/konkrotte/hydravista, a macOS booru-like browser that talks to a hydrus client, to the main Client API help

  • misc

  • if you right-click on a selection of multiple tags, you can now hide them or their namespaces en masse
  • if you right-click on a selection of multiple tags, you can now add or remove them from the favourites list en masse. if you select a mix of tags that are part-in, part-out of the list, you'll get both add and remove menu entries summarising what's going on. also, this command is now wrapped in a yes/no confirmation with full summary of what's being added/removed
  • the 'favourites' "tag suggestions" section is renamed to 'most used'. this was often confused with the favourites that sit under a tag autocomplete, and these tags aren't really 'favourite' anyway, just most-used, so they are renamed
  • if you have 'remove files from view when they are sent to the trash' set, then moving a file from one local file domain to another or removing one of multiple local file domains will no longer trigger a 'remove media'! sorry for the trouble, it was dumb logic on my part
  • fixed the 'known urls' menu's url class section ('open all blahbooru urls' etc...) not appearing when right-clicking a single 'collection' thumbnail
  • fixed the 'known urls' menu's open/copy specific urls not appearing when right-clicking any collection. it now shows the front 'display media's' urls
  • if you change the darkmode in _options->colours_, the _help->darkmode_ menu item now updates correctly. just a side note: I hate much of this system and will eventually unify it with the style system
  • fixed a bunch of 'number of x' tests at the database level when the operator is `≠`

  • system:number of urls

  • added `system:number of urls`! note this counts raw URLs at the moment--I just don't have fast database filtering of post urls vs file urls or url-classless urls or whatever. it does a raw count.
  • `system:known urls` is now tucked with this new `system:number of urls` under a new stub predicate called `system:urls`
  • a variety of 'system:number of words: has/no words' predicates now parse correctly when typed
  • wrote some new system predicate parsing tests

  • more cbz rules

  • cbzs' non-image files must now have an appropriate extension like .txt, .nfo, or .xml
  • the test regarding the count of non-image files (typically allowing up to 5 non-image files per directory) is more precise with regards to subdirectories, meaning a cbz with a single subdirectory and three non-image files now counts as a cbz
  • every cbz must now have at least two image files that contain a number of some sort

  • cleanup and boring stuff

  • I split the github workflow build file into three, so the windows, linux, and macOS builds now all happen and upload in parallel. previously, the upload step was blocked on the slowest of the three, which was typically the macOS build by about ten minutes; now they all upload whenever they are ready. this will also help some future testing situations. the newly split scripts are a little unclean/inefficient, so there is also more work to do here
  • I think I fixed the non-Windows executable permission bits for the various .sh and .command files in the base directory, which were lacking them, and I removed it from a couple dozen pngs across the docs and static directories, which somehow had them. let me know if I missed anything or messed anything up!
  • if you click one of the static system predicate buttons that appear in the system pred edit UI, for instance 'system:has duration', this no longer gets promoted to the 'recent' predicates list the next time you open the panel
  • some sytem predicate edit panels should stretch vertically a bit better
  • some 'number of tags' queries should be a little faster
  • the 'tag suggestions' options page has a bit of brushed up UI and some new explanation labels
  • unified the various thumbnail generation error reporting for all the different filetypes. it should also print the file's hash, too, since most of these error contexts only have a temporary path to talk about at this stage, which isn't useful after the fact

• version 557

  • misc

  • optimised large tag filter edit UI. you can now paste 5,000 items into an empty tag filter blacklist in less than a second, and if you have a big tag filter, removing or adding one thing is now instant (previously, this stuff would lag 4 seconds or more, sometimes multiple minutes!!)
  • the ugoira 'num frames' counting method now discludes files ending in .js/.json, to catch future bundling of frame timings
  • the cbz scanning tech should now recognise cbzs with four or fewer pages
  • a legacy 'is this image all good?' check that happens on PIL-loading is now gone. this improves rendering for a variety of truncated files and clarifies some error messages (previously, this thing was just failing silently)
  • fixed the delete file pre-flight logic so users on the non-advanced delete dialog can now delete repository updates. previously, they saw the menu entry, but hitting it was a no-op

  • better hash predicate parsing

  • `system:hash` labels are a little different now. they'll say `system:hash (md5) is abcd...`, with the algorithm after the "hash". hash is omitted for sha256 (the hydrus default). this eases parsing
  • `system:similar to data` labels are a little different. they'll say 'distance' instead of 'max hamming', and the number and type of hashes they hold, and if they hold only pixel hashes, the distance is not stated
  • `system:hash` predicate parsing is now more flexible. you can put the hash type pretty much anywhere now.
  • `system:similar to` and `system:similar to data` predicate parsing is now more flexible. more combinations are allowed, and you can not include distance and it'll be fine
  • these three hash predicates now copy to clipboard with all their hashes explicitly enumerated, making strings that are fully parsable! this is a big step forward in a completely sealed import-export predicate parsing loop; now I have the tech set up to export a different phrase to clipboard than what you see in the label, I just need the examples of where it goes wrong. if there is a system predicate that copies to clipboard in a way that won't parse back, let me know and I'll see if I can fix it.
  • added more unit tests for this parsing

  • documentation and cleanup

  • wrote a guide on how to install 'Git for Windows' for the 'running from source' help. although most of the settings in its marathon 12-page install wizard can be left as default, the technical questions can be intimidating, so I've written them all out for a nice simple install. also brushed up some of the surrounding help here
  • added a warning to the regular 'installing and updating' help regarding the danger of test-running extract releases before updating (you can overwrite your database by accident)
  • thanks to a user, the filetypes help document is updated with Ugoira and CBZ info
  • all the 'HydrusFiletypeHandling' files are refactored to a new 'files' module. there's a bunch of them these days!
  • the hydrus.core.images module is moved beneath this 'files' module too
  • the file log list panel right-click menu now says 'open URLs'/'open files' locations' depending on whether you are looking at a URL import log or local HDD import log

  • client api

  • the `file_metadata` call now returns `filetype_forced` and, if so, also `original_mime` to talk about the new forced filetype system
  • the client api help and unit tests are updated to test this is working ok
  • fixed a typo that was causing too much work in the updated file info manager call (and was often returning 'null' results for half-cached `file_metadata` requests with `only_return_basic_information=true`)
  • thanks to a user, the `/add_urls/get_url_info` Client API call now has a cache timeout of ten minutes, and the `/add_urls/get_url_files` call now has a timeout of 30 seconds if all the files are 'already in db'. this should automatically reduce some overhead for several programs that talk to the Client API a lot about URLs
  • the client api version is now 58

• version 556

  • misc

  • fixed, on a file drag and drop, the new export path eliding code from raising an error when the default export phrase would give an empty filename. e.g. if you set the export phrase as `[title]` and the file has no title. this no longer raises an error, and the fallback export phrase `{hash}` is again used instead. broadly speaking, most errors here are now handled better
  • also, export folders will now fallback to using `{hash}` if their normal export filename raises an error
  • holding down ctrl+shift+ while selecting thumbnails now does the same thing as a bare shift+ select. previously, it was unhelpfully interpreting this as a bare ctrl+ click
  • I may have improved the stability of 'minimise to system tray'. this thing still hangs the UI for some users on a delayed restore, I do not for certain know why
  • thanks to a user who figured out the new build script, the Docker package is now on Alpine 3.19, with more and newer python library support along with it

  • forced filetypes

  • you can now force files' filetypes. hit _right-click->manage->force filetype_ on thumbnails or the media viewer, and you'll get a new dialog that lets you force-reassign those files to be considered something else. changes take place immediately, and files are renamed on disk with their new file extensions, making 'open externally' work nicely. the original filetype is remembered, so this can be undone easily through the same dialog
  • this is happening because of the cbz/zip/Ugoira work, where the distinction between one format and another is not always perfect. the tech will also be useful for 'arbitrary file import' support. in any case, if there is something you want to force one way or another, it should now be easy
  • searching for system:filetype will recognise the forced filetypes, but there may be other, more advanced areas of the program that should but do not. please let me know how you get on!
  • there is a new system predicate, `system:has/no forced filetype`, that lets you further filter for the files that have this set or not. it is under `system:file properties`. it is also parsable if you ever need to type it
  • if a file gets a metadata rescan and becomes a different filetype, this affects the original filetype and not the forced. if they are now both the same, no big problem
  • as a side thing, I cleaned up how file metadata is put together in the database during file search. we were in a limbo state a little while ago, with an api call that just needed limited data, but I was never comfortable with it. now everything goes through the same routine, and every 'file info manager' is fully fleshed out, no matter the caller
  • _yes, if you set a zip as a jpeg, you are going to get weird errors when you click on them. I'll iron these things out a bit--and have already added several quick safety checks for apparent image files without resolution and so on--and I am interested in reports, but for the most part, don't be stupid here and you won't end up in a bad place_

  • filetypes

  • **you will be asked on update if you would like to regenerate all your animated GIF and APNG thumbnails. The new x%-in and transparency tech seems to be working well, so I'm rolling out the full regen to everyone**
  • before verifying a zip is an Ugoira or a cbz, the client now test-reads the cover page it will use as a thumbnail just to make sure it isn't passworded or corrupt or whatever
  • thanks to a user, the test for whether a a zip is encrypted is much faster and neater now
  • if there is an obvious video in a zip file, this is now dispositive to it not being considered a cbz
  • all cbz and Ugoira are going to get a metadata scan again to account for these stricter rules

  • Mr. Bones/file history chart

  • **if you have had some dodgy inbox/archive numbers in your file history chart, please check again and let me know what you see. if the numbers are still bad, try changing the search from the 'all my files'/'system:everything' default--any better?**
  • fixed Mr. Bones undercounting deleted files on some very old clients (i.e. mine)
  • improved accuracy of some archive/inbox time calculations for the file history chart by adjusting archive times to the file service removal time of that file, if it is earlier
  • included some additional de-inbox events that were being missed in the file history chart by recognising that files in the inbox but removed from a domain are nonetheless a decrement to the inbox count
  • on update, some old invalid archive records will be deleted, which will also help the file history chart

  • boot error handling

  • if you start the program with client.db/server.db but missing any of the auxiliary databases, the program now stops you before the new file creation starts with a blocking message saying what has happened. it advises whether you should quit the process now to diagnose the hard drive fault or attempt to continue with reconstruction
  • if you start the program with client.db/server.db but the 'version' table is missing, you now get a special blocking message before the main db creation routine starts saying what has happened. it advises whether you should quit the process now to diagnose the hard drive fault or attempt to continue with initial creation
  • the server gets a bit of 'safe blocking show message' tech this week, which prints this info to the console and asks for the user to hit enter to continue

• version 555

  • Ugoira/CBZ/Zip

  • the Ugoira/CBZ conversion last week went ok! we found too many false-positive Ugoiras, however, so I have decided to make that test stricter. Ugoiras now have to have zero-indexed filesnames, and always zero-filled to six digits. all your Ugoiras will get scanned again to see if they should better be CBZ
  • all zip files that are not openable (passworded, corrupt) are now detected early and just set as 'zip'

  • OpenCV

  • after discussing it with users, I have made the decision to slowly remove the image library OpenCV from the program. it has served us well, but it has always been a difficult-to-install bloat, and the super-compatible PIL actually does the job better these days. we'll simplify our rendering pipeline while also, with luck, improving HDR format support in future
  • thanks to a user, a critical OpenCV call involved in generating similar-files search metadata (perceptual hashes DCT) is now replaced with non-OpenCV tech
  • PIL can now load images in int32 or float32 greyscale, with or without ICC Profiles, and it shouldn't look too crazy (OpenCV was handling these before)
  • deleted all the old OpenCV gif rendering and metadata scanning tech
  • **if you would like to help test, please turn on `options->media->IN TESTING: Load images with PIL`. this used to be just a BUGFIX thing, but now it emulates where we actually want to end up. please send me any image files that render weird**

  • better boot error handling

  • if an error happens very early during boot, before the main Application event loop and splash screen are started, hydrus will now try and spin up a very small App and text dialog to show you the error visually! of course, if the error is Qt-related, then this won't work, ha ha ha, but you'll still always get the crash log
  • the client will now boot if the 'already-running' file exists but is incomprehensible--it'll just log that it was. also, if any other problem occurs during the 'already-running' check, hydrus assumes it is not already running and prints the error to the log
  • improved the 'can we write to the database folder?' test a little more. previously, if the db directory on boot was both missing and its parent was read-only, it would raise an error. now we correctly recognise that state as 'not writeable'
  • also, the fallback to the userpath db directory now only happens if you do not set a `-d/--db_dir` launch parameter. if you specifically set a launch path and that place is missing or read only, the program will not boot! I am more comfortable doing this now that we have the dialog to better display what happened
  • unified the 'what db dir are we using?' tests to one place
  • also cleaned up some of the boot failure code, which was spamming things haphazardly

  • string splitting and joining

  • the String Splitter and Joiner now interpret `\n` in their splitter/joiner text as newline (and other replacements like `\t` for tab; anything python supports). in order to not break existing parsers, the old splitter and joiner strings will be encoded on update (any `\` will become `\\`)
  • added some unit tests to test this behaviour for both String Processor types

  • misc

  • the system predicate parser is now plugged into the excellent `dateparser` library that we already use in downloader parsing. this thing can eat pretty much any date string you can throw at it, so if you type "system:archived time: since 01/05/2011" or "system:archived time: before 30 hours ago", it'll all work for almost any combination you can think of. it'll probably even work in your native language! the one big caveat is if you give a longer duration timestamp in the form 'x time units( ago)', rather than a specific date, it'll convert it to days/hours, ignoring years and months. since this stuff causes a ton of headaches, I am likely going to switch all the time-delta time predicates here to work on days/hours/seconds, and if you want to put 60 or 365 days, knowing what inaccuracy that implies means, then you can, rather than have me continually fret over and fail to deliver various leap year calculation problems. _calendarium delenda est_
  • fixed some thumbnail rendering for another class of damaged gif--this time, gifs that are so garbagified that they change their resolution from one frame to the next and/or produce a sizeless, shapeless frame of a handful of bytes. this is now detected and the bad data discarded!
  • if a video seems to have 0/None duration, the main native ffmpeg renderer (which is also used for thumbnail generation) can now handle it. the 'start x% in' value will be crazy, but it'll work
  • fixed an error with mpv trying to inspect the duration of null media during various states of media viewer transition

  • boring cleanup

  • gave a quick pass over the ~250 small 'just show some text and a system icon' dialogs work across the program. unified all calls through one location, improved some strings and string formatting, added more exception logging, unified the dialog titles, differentiated information/warning/critical flags better, made 'critical' messages log their titles and text, and made it all thread safe in a nice invisible way to callers
  • fixed some borked page/popup permission checks in the client api
  • if a file transitions from 'no transparency' to 'has transparency', the client will now queue a thumbnail regen, just in case that tech has been recently added
  • improved the formatting of what the main error-logging method actually prints to the log
  • slimmed down some of the watcher/subscription fixed-checking-time code
  • misc formatting cleanup and surplus import clearout
  • fixed the discord link in the PTR help document

• version 554

  • checker options fixes

  • **sorry for any jank 'static check interval' watcher or subscription timings you saw last week! I screwed something up and it slipped through testing**
  • the 'static check interval' logic is much much simpler. rather than try to always keep to the same check period, even if the actual check is delayed, it just works off 'last check time + period', every time. the clever stuff was generally confusing and failing in a variety of ways
  • fixed a bug in the new static check time code that was stopping certain in-limbo watchers from calculating their correct next check time on program load
  • fixed a bug in the new static check time code that was causing too many checks in long-paused-and-now-unpaused downloaders
  • some new unit tests will make sure these errors do not happen again
  • in the checker options UI, if you uncheck 'just check at a static, regular interval', and leave the faster/slower values as the same when you OK, then the dialog now asks you if that is what you want
  • in the checker options UI, the 'slower than' value will now automatically update itself to be no smaller than the 'faster than' value

  • job status fixes and cleanup (mostly boring)

  • **sorry for any 'Cancel/IsCancellable' related errors you saw last week! I screwed something else up**
  • fixed a dumb infinite recursion error in the new job status cancellable 'safety' checks that was happening when it was time to auto-dismiss a cancellable job due to program/thread shutdown or a maintenance mode change. this also fixes some non-dismissing popup messages (usually subscriptions) that weren't setting their cancel status correctly
  • this happened because the code here was ancient and ugly. I have renamed, simplified, and reworked the logical dangerzone variables and methods in the job status object so we don't run into this problem again. 'Cancel' and 'Finish' no longer take a seconds parameter, 'Delete' is now 'FinishAndDismiss', 'IsDeleted' is now 'IsDismissed', 'IsDeletable' is now merged into a cleaner 'IsDone', 'IsWorking' is removed, 'SetCancellable' and 'SetPausable' are removed (these will always be in the init, and will determine what type of job we have), and the various new Client API calls and help are updated for this
  • also, the job status methods now check their backstop 'cancel' tests far less often, and there's a throttle to make sure they can only run once a second anyway
  • also ditched the needless threading events for simple bools
  • also cleared up about 40 pointless Finish/FinishAndDismiss duplicate calls across the program
  • also fixed up the job status object to do its various yield pauses more sanely

  • cbz and ugoira detection and thumbnails

  • CBZ files are now detected! there is no very strict standard of what is or isn't a CBZ (it is basically just a zip of images and maybe some metadata files), but I threw together a 'yeah that looks like a cbz' test that now runs on every zip. there will probably be several false positives, but with luck fewer false negatives, which I think is the way we want to lean here. if you have just some zip of similarly named images, it'll now be counted as a CBZ, but I think we'll nonetheless want to give those all the upcoming CBZ tech anyway, even if they aren't technically intended to be 'CBZ', whatever that actually means here other than the different file extension
  • the client looks for the cover image in your CBZ and uses that for the thumbnail! it also uses this file's resolution as the CBZ resolution
  • Ugoira files are now detected! there is a firmer standard of what an Ugoira is, but it is still tricky as we are just talking about a different list of zipped image files here. I expect zero false negatives and some false positives (unfortunately, it'll be CBZs with zero-padded numerical-only filenames). as all ugoiras are valid CBZs but few CBZs are valid ugoiras, the Ugoira test runs first
  • the client now gets a thumbnail for Ugoiras. It'll also use the x%-in setting that other animations and videos use! it also fetches resolution and 'num frames'. duration can't be inferred just yet, but we hope to have some options (and actual rendering) happening in the medium-term future
  • this is all an experiment. let me know how it goes, and send in any examples of it failing awfully. there is lots more to do. if things don't explode with this much, I'll see about .cbr and cb7, which seems totally doable, and then I can seriously plan out UI for actual view and internal navigation. I can't promise proper reader features like bookmarks or anything, but I'll keep on pushing
  • all your existing zips will be scheduled for a filetype re-scan on update

  • animations

  • the native FFMPEG renderer pipeline is now capable of transparency. APNGs rendered in the native viewer now have correct transparency and can pass 'has transparency' checks
  • all your apngs will be scheduled for the 'has transparency' check, just like pngs and gifs and stuff a couple weeks ago. thanks to the user who submitted some transparency-having apngs to test with!
  • the thumbnails for animated gifs are now taken using the FFMPEG renderer, which puts them x% in, just like APNG and other video. transparency in these thumbnails also seems to be good! am not going to regen everyone's animated gif thumbs yet--I'll do some more IRL testing--but this will probably come in a few weeks. let me know if you see a bevy of newly imported gifs with crazy thumbs
  • I also overhauled the native GIF renderer. what used to be a cobbled-together RGB OpenCV solution with a fallback to bad PIL code is now a proper only-PIL RGBA solution, and the transparency seems to be great now (the OpenCV code had no transparency, and the PIL fallback tried but generally drew the last frame on top of the previous, giving a noclip effect). the new renderer also skips to an unrendered area faster
  • given the file maintenance I/O Error problems we had the past couple weeks, I also made this cleaner GIF renderer much more robust; it will generally just rewind itself or give a black frame if it runs into truncation problems, no worries, and for gifs that just have one weird frame that doesn't break seek, it should be able to skip past those now, repeating the last good frame until it hits something valid
  • as a side thing, the FFMPEG GIF renderer seems capable of doing almost everything the PIL renderer can now. I can flip the code to using the FFMPEG pipeline and gifs come through fine, transparency included. I prefer the PIL for now, but depending on how things go, I may add options to use the FFMPEG bridge as a testbed/fallback in future
  • added some PIL animated gif rendering tech to handle a gif that out of nowhere produces a giga 85171x53524 frame, eating up multiple GB of memory and taking twenty seconds to failrender
  • fixed yet another potential source of the false positive I/O Errors caused by the recent 'has transparency' checking, this time not just in malformed animated gif frames, but some busted static images too
  • improved the PIL loading code a little more, converting more possible I/O Errors and other weird damaged file states to the correct hydrus-internal exception types with nicer error texts
  • the 'disable CV for gifs' option is removed

  • file pre-import checks

  • the 'is this file free to work on' test that runs before files are added to the manual or import folder file list now has an additional file-open check. this improves reliability over NAS connections, where the file may be used by a remote process, and also improves detection for files where the current user only has read permissions
  • import folders now have a 'recent modified time skip period' setting, defaulting to 60 seconds. any file that has a modified date newer than that many seconds ago will not be imported on the current check. this helps to avoid importing files that are currently being downloaded/copied into the folder when the import folder runs (when that folder/download process is otherwise immune to the existing 'already in use' checks)
  • import folders now repeat-check folders that have many previously-seen files much faster

  • misc

  • the 'max gif size' setting in the quiet and loud file import options now defaults to 'no limit'. it used to be 32MB, to catch various trash webm re-encodes, but these days it catches more false positives than it is worth, and 32MB is less of a deal these days too
  • the test on boot to see if the given database location is writeable-to should now give an error when that location is on a non--existing location (e.g. a removable usb drive that is not currently plugged in). previously, it could, depending on the situation, either proceed and go crazy later or wait indefinitely on a CPU-heavy busy-wait for the drive to be plugged back in. unfortunately, because at this stage there is no logfile location and no UI, if your custom db dir does not and cannot exist, the program terminates instantly and silently writes a crash log to your desktop. I have made a plan to improve this in future
  • also cleaned up all the db_dir boot code generally. the various validity tests should now only happen once per potential location
  • the function that converts an export phrase into a filename will now elide long unicode filenames correctly. filenames with complex unicode characters will take more than one byte per character (and most OSes have ~255 byte filename limit), which requires a trickier check. also, on Windows, where there is a 260-character total path limit, the combined directory+filename length is checked better, and just checked on Windows. all errors raised here are better
  • added some unit tests to check the new path eliding tech
  • brushed up the 'getting started with ratings' help a little

  • client api

  • thanks to a user, the Client API now has the ability to see and interact with the current popup messages in the popup toaster!
  • fixed a stupid typo that I made in the new Client API options call. added a unit test to catch this in future, too
  • the client api version is now 57

• version 553

  • animated gif fixes

  • fixed the **false positive** "serious I/O Error! This is a significant hard drive problem" problems saw in last week's animated gif 'has transparency' rescanning. it turns out the PIL animated gif renderer we rarely use was raising overly serious errors on a truncated frame (i.e. some borked file, not a borked OS hard drive access), and this was escalating up to the overall maintenance system, which was shutting down in panic. truncated gifs in the PIL renderer should now just either render the last frame over and over or rewind as soon as they hit a super problem
  • as well as rendering, the duration and frame-counter code also handles these borked frames better, so animated gifs that have a single borked frame should now A) import without an error and B) get more accurate frame counts
  • you may have seen this sort of error before where an mpv window seems to keep rendering the gif despite the scanbar hitting its right end. if you see a file like that, try right-clicking and hitting _manage->maintenance->regenerate file metadata_. might just fix it up!
  • if mpv encounters one of these busted gifs, a selection of quiet-but-spammy "I don't know what happened, but MPV just reported a weird error, here it is" logging no longer happens. we basically know what happened, and mpv seems good at recovering
  • fixed some PIL alpha gif rendering on backwards seek and loop

  • slideshow tech

  • _options->media_ has a new 'slideshow' section with five new options regarding slideshows and media with duration (video, audio). if you don't care too much, just leave them alone--they make slideshows transition better!
  • first, there's a checkbox to say 'always play duration-having media completely once through before moving on'. this was the previous behaviour, now default off
  • then there are two options, for a percentage of the current slideshow period and a flat seconds value, to say 'if the duration-having media is shorter than this amount of time, then move the slideshow on early'. this allows short gif loops to play a bit, but not for 15 minutes
  • then there is an option, for a percentage of the current slideshow period, to say 'if the duration-having media has a duration between this amount and 100% of the slideshow period, then move on once it has played once through'. this allows for clean slideshow transitions for media that is just a bit shorter than the slideshow period
  • then there is an option, again for a percentage of the slideshow period, to say 'if the duration-having media is _longer_ than the slideshow period plus this amount of time, then delay moving on until it is played once through'. this allows 35 second videos to complete fully in a 30 second slideshow while stopping a ten minute vid hogging its turn
  • completely rewrote the slideshow timer tech
  • did a little work bringing the experimental Qt media player up to proper slideshow capability, and neatened the associated code
  • yes, hydev did write all these options for his repurposed slideshow computer because he was annoyed about his vidya captures and 500ms loops playing jank on a 30m slideshow period

  • misc

  • the file-info-summary lines that appear in the top row thumbnail menu submenu now show if a file has audio/transparency/exif/other metadata/icc profile
  • the file-info-summary lines that appear in the top row thumbnail menu submenu and the top-center of the media viewer no longer list 'removed from x 5 days ago' for files that were moved internally between local file services. these statements were spammy and not helpful! if you really need them, are available in the 'manage times' dialog. sorry for the annoyance here
  • trying to move a file from local file service x to y no longer triggers the archive delete lock, if you have that enabled (this was prohibiting the delete from x after the copy to y is done)
  • the 'import options' button now labels itself with 'all default', 'all set', or 'some set' to quick-review what it is holding
  • the stupidly named advanced users' `OR*` button is now just `advanced`
  • the 'manage->regenerate' thumbnail menu is now called 'maintenance', and it always contains the whole file maintenance job list just as it appears in the main file maintenance panel. tooltips are the longer descriptions
  • if you reduce a static check time in a checker options (watchers or subscriptions), the next check time should now recalculate correctly immediately. previously, the new check period wasn't kicking in until the next, delayed cycle. I have preserved the logic that tries to keep a static check time regular (which was the core problem here), where if you check every 7 days on saturday night, then delaying one time and running it on sunday night won't delay the check time phase along to next sunday--the next week it will be due on saturday again
  • the system:rating edit panel has some tooltips that say 'Set "is" and leave rating null to search for "unrated".'. maybe this is annoying, maybe I should just add redundant 'unrated' checkboxes, let me know what you think
  • when the file maintenance routine runs into a serious error (like we had with the false positive transparent gifs), the popup messages now include a file button for the problem file for easy referral
  • if a file breaks MPV in a crashy-looking way, hydrus now makes a popup with a file button to it for easy referral
  • fixed a typo issue with the recent temp folder recovery code that broke the temp folder for file imports on the hydrus file repository. sorry for the trouble, this slipped through unit testing because that too has a hacky temp folder solution!

  • weird/specific stuff

  • I've spaced out many of the initial library loads across the core boot init routine. fingers crossed, the splash screen will open earlier and report more as things each import, rather than taking ages to appear and then suddenly initialising everything real quick (on slow computers). also, a monolithic UI init job is broken up into pieces, which should let the splash update itself a little smoother as your client loads its style and stuff
  • the file and database maintenance managers are now initialised in a better stage, and, like the subscription manager, they now do not start any work until the first session is fully loaded
  • while pouring over the server code trying to find a petition miscount bug and/or a petition summary fetch bug (which I was entirely unsuccessful at), I stopped it 404ing when there are unexpectedly no petitions to fetch--it should now just give an empty list and reset the count serverside, which is an old behaviour that wasn't working quite right in the modern system. this will cost less CPU than commanding the full service service_info number reset. I will have to investigate the core problem of miscounts more closely to figure out the base problem here
  • if two subscription jobs publish files to the same popup label (which merges the popup button to include both sets of files), this file list is now properly deduplicated (so if both subs picked up the same file(s), it now won't try to publish the same file twice). the basic popup button instantiation also clears out dupes
  • the 'additional tags'-only tag import options in a subscription query can now no longer be set 'default'. this choice was unintended and has no current meaning if set
  • building on last week, there are even fewer duplicate menu tooltips
  • updated and clarified the text in _options->external programs_. also, if you try to put a command that does not include "%path%", it'll moan at you with a yes/no dialog
  • when 'confirm sending files to trash' is off but 'use advanced file delete dialog' is also on, the advanced file delete dialog will now not pop up if there is only one normal local file service to delete from (it was always popping up before, since it exposes the physical file delete options and the dialog thought it wasn't a 'one-choice' delete)
  • the forced-wait throttle that happens on several exception catches is reduced from 1s to 200ms
  • I made the new job status queue properly thread-safe with a lock. I forgot to do it last week, whoops!
  • fixed the build script to construct a file named .tar.zst for the Ubuntu release, not .tar.gz

• version 552

  • misc

  • 'system:has audio' and 'system:embedded metadata' are now combined under a new meta-system predicate 'system:file properties'. if you can't find your yes/no predicate, try looking there!
  • menu commands will no longer have their unadjusted label as their tooltip. all tooltips are either the full status bar description or the full label if it was long enough to be elided
  • the 'open externally' panel now shows the default filetype thumbnail for formats like zip and epub
  • 'system:number of character tags > 4' now parses correct when you type it (previously it wouldn't work with a namespace), including special handling for 'unnamespaced'
  • the various 'number of x' system predicates will now parse if you type 'num x', 'number x', or 'num of x'
  • to match the other entries, the '4k' resolution swap-in label is now '2160p'
  • added a little extra info on the manage tags dialog to 'getting started with tags'
  • if you have 'confirm sending files to trash' turned off, the delete dialog will now show on physical deletes (i.e. deletes from the trash)
  • updated the derpibooru parser to pull the new AI-based 'generator' and 'prompter' namespaces (converting both to the hydrus-appropriate 'creator')
  • thanks to a user, the Linux build is now archived with zstd instead of gzip. should be about the same size but faster to decompress

  • fixes

  • fixed a stupid typo in the folder copy/move tech last week that was not allowing some move/copies to start (as always, the thing that is so simple that you don't think to test it is the very thing that blows up). sorry for the trouble!
  • cleaned up the file/folder move/copy error statements a little more
  • fixed the 'default search page tag service' dropdown in _options->search_ not saving correctly
  • fixed the 'open externally' panel having out of position thumbnails when your thumbnail supersampling is set to other than 100%
  • fixed the import and display of images in signed 16-bit format (weird TIFFs, seems like).
  • any image with an unusual channel data type beyond uint16 and int16 is going to be, as the default thing to do, normalised to unsigned 8-bit. it may blow out the colour range, but it should show something!
  • the client handles files with (0x0) resolution better. they should now always import, and it'll _attempt_ to render them to a normal full size thumb. if it works (e.g. this is some misconfigured SVG), great, and if it doesn't, we'll get a nicely sized filetype.png or hydrus.png fallback
  • files with (0x0) resolution will now never show in the preview or media viewers. previously, the preview viewer would bail out half-way through setting the media, causing it to fall into an invalid state where it still showed the previous valid media but wouldn't 'click-off' it easily, and the media viewer would generally panic to its 'no media to show' state and lose navigation functionality. now, files that are 0x0 are included in the general 'can we show this?' pre-launch sanity checks

  • has_transparency

  • the database can now remember if a file has transparency. you can search this with the new 'system:has transparency' predicate, which is under the new 'system:file properties' and will also parse if you type 'system:has/no transparency/alpha'
  • note that my version of 'has transparency' discludes files that have an all-opaque alpha channel (i.e. one that lets no light through). RGBA is insufficient--I want an alpha channel with some actual translucency somewhere!
  • although many application image project types like PSD and XCF can have transparency, the various ways we render or thumbnail them are hacky and probably lock to RGB or RGBA always, so I'm going to start simple. this week, we test transparency for all the images that support it (basically anything but jpeg), and animated gif. the animated gif tech is new and actually looks through every frame of an RGBA gif until it hits interesting alpha to catch cases where it starts opaque and fades away
  • just like we had with 'has exif' and similar, 'has transparency' knowledge will be calculated instantly for all new files, but for the files you already have, we'll have to do some slow file maintenance in the background for a while to retroactively calculate it all. you don't have to do anything; the data will just populate over time
  • the duplicate filter now shows 'has transparency, the other is opaque' statements
  • while working on this, I encountered a number of files that seemed to be false positives--apparently normal, fully opaque images of anime girls that were somehow showing up as 'has interesting alpha'. upon inspecting them closely, I discovered the border pixels had a slight fade, or one pixel out of all of them was 98% opaque, or the single bottom right pixel was completely transparent. perhaps some of these are secret artist markers, but I imagine many are just an accidental drawing tablet smudge or dodgy crop tool calculations. I'm leaving them as 'has_transparency' for now, but maybe we'll want to tune this more in future, perhaps saying you have to be at least 0.3% transparent to count. anyway, as always, while I am interested in seeing files that seem to get a false positive/negative with this new 'has transparency' test, if you have the technical know-how, please check if they actually have no alpha yourself first. once you play around with this system, let me know what sort of pseudo-'false positive' rate you are getting, and we can talk about an appropriate threshold

  • client api

  • the 'file_metadata' call now includes a 'has_transparency' boolean! remember that it will be overly `false` for a while, until the file maintenance catches up
  • forgot to mention it last week, but thanks to a user there is a new `/manage_database/get_client_options` call that fetches a heap of different client options. this exposes a mess that may change with any update, but there may be something neat you can hook into. this week we fixed a thing that was breaking this call for probably all old clients
  • the client api version is now 56

  • boring cleanup

  • renamed JobKey to JobStatus across the program
  • in prep for Client API calls to interact with the popup system, the queue of JobStatuses waiting to be displayed in the popup toaster is now encapsulated in a separate class, outside of the Qt object dangerzone
  • sped up how the popup manager system inspects and cleans the JobStatus queue in general. should have better performance when you get hundreds or thousands of messages
  • cleaned up some awkward popup manager dismiss code
  • fixed a timing issue that meant popup messages were auto-dismissed from the popup toaster up to a second after they were being 'deleted' by their parent functiions. subscription flow felt more laggy because of this
  • fixed the file info manager's duplicate call to duplicate unusual metadata like has_exif and blurhash
  • removed some old code that isn't used any more

• version 551

  • misc

  • thanks to a user, we have a new checkbox under _options->thumbnails_ that disables thumbnail fading. they'll just blink into place in one frame as soon as ready
  • after looking at this code myself, I gave it a full clean. the actual thumbnail fade animation is now handled with some proper objects rather than a scatter of variables passed around
  • I also doubled the default fade time to 500ms. I expect I'll add an option for it, especially if we rework all this into the proper Qt animation engine and get it performing better
  • fixed the crashes users on PyQt were seeing! I made one tiny change (1->1.0) last week, and PyQt didn't like it, so any view of Mr Bones or 'open externally' panels, or the media viewer top-right ratings hover was leading to program instability
  • the system predicates for 'has/no duration', 'has/no frames', 'has/no notes', 'has/no words' (i.e. the respective 'num x' system pred, but either = 0 or >0) are now aware that they are each others' inverse, so if you ctrl+double-click or do similar edit actions, they'll flip
  • updated the 'PTR for dummies' page to link to a new QuickSync source, kindly maintained and hosted by a user

  • code cleanup and misc bug fixes

  • sped up some random iteration across the program (e.g. when deciding which order to waterfall thumbnails in, which can suffer from overhead if you do a fast giganto-scroll)
  • cleaned up the code that does image alpha channel (transparency) detection, comparison, and stripping
  • unified how the variety of image loads and conversions perform the 'strip this image of useless transparency data' normalisation step. thumbnails from krita, svg, and pdf are now stripped of useless alpha. also, all 'import this serialised object png' avenues now handle pngs with spurious alpha
  • I think I fixed the alpha channel stripping code to handle 'LA' (greyscale with transparency) files. if you try to import a hydrus serialised object png file that is for some crazy reason now LA, I think it'll work!
  • when a files popup message filters its current files and the count goes to 0 (happens if you re-click the button after deleting everything it has to show), the message now auto-dismisses itself (previously it was nuking the button but staying as a thin strip of null panel space)
  • fixed a bug where `system:date` predicates were displaying labels an hour off (usually midnight -> 11pm, thus cycling back to the previous day) thanks to the clocks changed (in the USA) last weekend. I suspect there is more of this, here and there, so let me know what you see
  • fixed a counting typo error with the delete files code when you delete the last file in a domain but the domain thinks it already has 0 files
  • fixed up similar code across the database to forestall future typos on SQLite SUMs
  • improved and unified the 'hydrus temp dir' management code. if the specific per-process hydrus temp dir is cleared out by an external factor (I'm guessing just the OS cleaning up during a long running client session), hydrus should just simply make a new folder as needed. with luck, this will fix a problem with drag and drop export that ran into this

  • many file move/copy error handling improvements

  • _tl;dr: if hydrus can't put a file somewhere, it deals with that better now_
  • improved how file move/merge function reports its errors, and how all its callers handle them
  • the 'rename a file's file extension when its filetype changes' job now correctly recognises when it fails to rename a file due to a reason other than the file being currently in use
  • import folders now correctly detect when they fail to 'move' action a file out after processing
  • the check file integrity routine now correctly detects when it fails to move a damaged file from file storage to a landing zone in the main db directory. this failure now cancels the job properly and prints a nicer error to the log
  • improved how the file copy/mirror function reports its errors, and how all its callers handle them
  • saving a serialised object png now properly catches a 'transfer from temp dir to dest location' move error
  • the internal database backup and restore routines now detect file copy errors better
  • a drag and drop export operation that wants to put the files in the temp dir and also fails to collect its files nicely now correctly raises an error
  • failing to set the mpv file on options save (and the subsequent mpv-load action) now reports its error correctly
  • exporting update files now handles a missing update file more gracefully
  • mergedirectory and mirrordirectory now fail instantly after any single error, rather than several
  • added some more file/directory pre-checks to all the merge/mirror functions
  • deleted some old unused code here

  • client api

  • thanks to a user, the Client API now has a 'generate_hashes' endpoint that returns the sha256 hash (and pixel hash and perceptual hashes of any appropriate image file) of any file you give it
  • the client api version is now 55

• version 550

  • misc

  • if you enter invalid URLs (i.e. non-parsing) into 'manage URLs', the dialog now lets you know they were not apparently good and asks if you want to enter them anyway. previously, it errored-out and disallowed anything that wasn't parsing ok (issue #1444)
  • when physically deleting files (i.e. deleting from trash or picking 'permanently delete' from the advanced delete dialog), the relevant files are now immediately removed from view. there were some situations where, when physically deleting a lot of files (causing the job to clear in batches), you could subsequently click on a soon-to-be-deleted file, loading it in mpv, and then, if you started a big UI-lag job like loading 'manage siblings', it could cause a crash if the file was deleted during the UI hang (issue #1447)
  • the client now explicitly closes and clears its network connections after five minutes of inactivity. it turns out that the behind the scenes tools were not doing this exactly as I had thought, clogging up connection slots (issue #1458)
  • thanks to a user, the rendering of palettized PNGs with ICC profiles is fixed!
  • fixed the github build script to include the new-as-of-a-couple-of-weeks-ago 'auto_update_installer.bat' file in the Windows builds. sorry for the confusion here, I forgot I had to do this!
  • optimised deselection of a large number of files when you already have a lot of thumbnails selected (a tricky example of this is clicking on an unselected file when you have a lot of files selected, thus deselecting all that old stuff). should be a little faster to work on big lists now
  • further optimised reduction recalculation of the taglist in general

  • thumbnail fill

  • after vacillating and talking about it for months, I finally reworked how ''scale to fill' thumbnails work. as sometimes happens, I only had to change about six critical lines of code to get the core functionality changed and nothing seems to have exploded
  • the main change here is KISS--'fill' thumbnail image files on disk are no longer clipped to just the viewable area, but the whole image scaled to fill the thumbnail space (with exceptions for extreme cases). this change gives us some simplicity and flexibility behind the scenes, saves some regeneration work when the user only changes one thumbnail dimension setting, improves maintenance tasks based off the thumbnail (like blurhash), and means that the Client API can fetch your thumbs and still have something useful to display
  • if you have 'scale to fill' set, hydrus will regenerate your thumbnails naturally as you browse the client. fingers crossed, you won't notice any visual difference through the transition
  • 'open externally' button panels now display their thumbnails with more reasonable maximum dimensions, and when things are gonk for whatever reason, they should nonetheless be centered correctly
  • as a side thing, this change allowed me to finally purge all the clipping tech from the thumbnail pipeline, where it had obtusely sunk in to every possible filetype thumbgen

  • eager login system

  • I fixed a problem where some sorts of login script could allow a network job supposedly waiting on them to start before they had completed. it was due to a complicated 'am I logged in?' cookie testing issue while the login process was still working. all network jobs that hypothetically need a login now test if there is a login process currently working on their domain and will properly wait for that process to finish before they move on
  • fixed a 'cannot log in' reporting bug in the login system
  • some misc login code cleanup

  • smarter orphan file record and repository update handling

  • _this is advanced stuff, most users can ignore_
  • _database->db maintenance->clear orphan file records_ is now able to recover file records where A) the file is in a service component but not the master, B) the file exists on disk. it copies the import timestamp from the specific to the umbrella domain and spams all the repaired files to a new page for user review. this maintenance routine isn't used all that much, but when you have a damaged database, it is nice to recover as much as possible rather than having to export (with clear orphan file records+clear orphan files) and then reimport and lose archive/inbox status and import timestamps
  • repository update files now have a 'delete from repository updates' entry in their right-click menu
  • this area of the code appears to be related to the PTR 404 issue some users have had (it seems to be repository update records not beeing added/deleted/updated correctly), so I am likely to revisit this
  • deleting a file from 'all local files' (which happens for repository update files) now correctly updates the UI-level media object to recognise that the file is fully deleted from all local file domains beneath the umbrella, removing the 'delete from x' commands from their menu, and in the right view contexts removing them from view completely

• version 549

  • misc

  • optimised taglist sorting code, which is really groaning when it gets to 50k+ unique tags. the counting is more efficient now, but more work can be done
  • optimised taglist internal update recalc by updating existing items in place instead of remove/replace and skipping cleanup-sort when no new items are added and/or the sort is not count-based. it should also preserve selection and focus stuff a bit better now
  • thanks to a user, we have some new url classes to handle the recent change in sankaku URL format. your sank subscriptions are likely to go slightly crazy for a week as they figure out where they are caught up to, sorry!
  • if a file copy into your temp directory fails due to 'Errno 28 No space left on device', the client now pops up more information about this specific problem, which is often a symptom of trying to import a 4GB drive into a ramdisk-hosted tempdir and similar. many Linux flavours relatedly have special rules about the max filesize in the tempdir!

  • maintenance and processing

  • _advanced users only, don't worry about it too much_
  • the _options->maintenance and processing_ page has several advanced new settings. these are all separately hardcoded systems that I have merged into more of the same logic this week. the UI is a tower of spam, but it will serve us useful when we want to test and fine tune clients that are having various sorts of maintenance trouble
  • a new section for potential duplicate search now duplicates the 'do search in idle time' setting you see in the duplicates page and has new 'work packet time' and 'rest time percentage' settings
  • a new section for repository processing now exposes the similar 'work/rest' timings for 'normal', 'idle', and 'very idle' (after an hour of idle mode). **if I have been working with you on freezes or memory explosions during PTR processing, increase the rest percentages here to 50-2,000, let's see if that gives your client time to breathe and clean up old work**
  • a new section for sibling/parent sync does the same, for 'idle', 'normal', and 'work hard' modes **same deal here probably**
  • a new section for the deferred database table delete system does the same, for 'idle', 'normal', and 'work hard' modes
  • I duplicated the 'do sibling/parent sync in idle/normal time' _tags_ menu settings to this options page. they are synced, so altering one updates the other
  • if you change the 'run file maintenance jobs in idle/normal time' settings in the dialog, the _database_ menu now properly updates to reflect any changes
  • the way these various systems calculate their rest time now smoothes out extreme bumps. sibling/parent display, in particular, should wait for a good amount of time after a big bump, but won't allow itself to wait for a crazy amount of time

  • all deleted files

  • fixed the various 'clear deletion record' commands to also remove from the 'all deleted files' service cache, which stores all your deleted files for all known specific file services and is used for various search tech on deleted file domains
  • also wrote a command to completely regen this cache from original deletion records. it can be fired under _database->regenerate->all deleted files_. this will happen on update, to fix the above retroactively
  • removed the foolish 'deleted from all deleted files' typo-entry from the advanced multiple file domain selector list. the value and use of a deletion record from a virtual combined deletion record is a complicated idea, and the entities that lurk in the shadows of the inverse sphere would strongly prefer that we not consider the matter any more

  • running from source stuff

  • **the setup_venv script has slightly different Qt questions this week, so if you have your own custom script that types the letters in for you, double-check what it is going to do before updating this week!**
  • there's a new version of PySide6, 6.6.0. the `(t)est` Qt version in the 'setup_venv' now points to this. it seems fine to me on a fairly normal Win 11 machine, but if recent history is any guide, there's going to be a niggle somewhere. if you have been waiting for a fix on the menu position issue or anything else, give it a go! if things go well, I'll roll this into a larger 'future' test release and then we'll integrate it into main
  • also, since Qt is the most test-heavy library we have, the 'setup_venv' scripts for all platforms now have an option to `(w)rite` your own version in!
  • the program no longer needs `distutils`, and thus should now be compatible (or less incompatible, let's see, ha ha) with python 3.12. thanks for the user report and assistance here

  • boring stuff

  • rejiggered a couple of maintenance flows to spend less time aggressively chilling out doing nothing
  • the hydrus timedelta widget can now handle milliseconds
  • misc code cleaning
  • fixed a typo in the thumbnail 'select->local/not local' descriptions/tooltips

• version 548

  • user contributions

  • thanks to a user, krita files are now renderable! we've got the defaults set like psds for now, where the preview viewer will show 'open externally', but the media viewer tries to load the full thing. let's see how it goes, and as always, if you have one that doesn't work, please send it in! note that krita are now eligible for the similar files system, so I've queued them up to get entered into it
  • thanks to a user, setting an IPFS 'nocopy' path including your home directory (~) should now expand correctly (issue #1320)
  • thanks to a user, newly-IPFS-pinned files are properly aware of their multihashes now (previously you needed a client restart or media reload after a delay) (issue #1328)
  • thanks to a user, the url and hdd downloaders now have 'stop/abort' buttons, which will stop current work and cancel the rest of the queue. I added a yes/no dialog where you can choose to skip or delete the remainder of the queue and a couple of bells and whistles like disabling the button when the current queue has no remaining work

  • misc

  • fixed an issue with successive drag and drop file exports that gave different files the same filename. previously, the successive files were being replaced with the first instance with the shared name (basically the original files were not being 'overwritten'), but it should be fixed now!
  • various places that were sorting services pseudorandomly now do so alphabetically (the F9 new page selector was doing this with local file domains (the first buttons in 'file search'), if you had multiple set up. sorry if I mess with your muscle memory here, but things should be more reliable here going forward!)
  • added a first version of an auto-update script, `auto_update_installer.bat`, to the main install directory. it will download the latest Windows exe installer using winget and install it to the current location. if you use the installer, you might want to experiment with it (make a backup first!) as an easy hands-free update solution. let me know how it goes, and if there are no problems in a couple of weeks, I'll add it to the help
  • added some more mpv error handling. if the mainloop behind your mpv window halts (which happens on various internal problems), we now detect it and more gracefully disable the viewer and its commands (previously it would escalate to error popups and try to keep working)
  • fixed an issue in the newer 'missing file storage recovery' code if there is more than one base location missing

  • thumbnail shortcuts

  • I converted all the old hardcoded thumbnail keyboard shortcuts (thumbnail focus movement, open-media-viewer, and select-files) to the newer user-editable system under _file->shortcuts_, under a new set called 'thumbnails'. there are some new file-filters too, so you can set up 'select inbox' and similar beyond the default ctrl+a to 'select all' and escape to 'select none'
  • I don't expect many people will want to even touch the giganto list of (shift+)(numpad)left/right/up/down/page up/page down/home/end selection combinations, but if you want to, you can!
  • the thumbnails set also now allows 'launch the archive/delete filter', which had an odd home in 'media' before. new users now start with F12 set up in 'thumbnails', not 'media'
  • I removed the jank semi-secret 'ctrl+space' hardcoded 'deselect current focused thumbnail' shortcut. that tech will probably return when I figure out more sensible logic and user settings around shift+ and ctrl+ behaviour
  • this cleanup reduces three different shortcut handling routines down to one, and it particularly clears the last place where I was using ancient grandfathered wx-based 'accelerator table' tech. it should be easier to update the thumbnail shortcuts in future, and I hope to plug the mouse into it also, so you can edit middle-click to launch media etc..

  • client api

  • after much discussion and personal vacillating, I have decided to include the `version` and `hydrus_version` in every JSON Client API response. CBOR responses are not affected. if you need to hook into these numbers for a completely stateless interface, it is now super convenient. I'm not delighted with the spamminess of this, but it is just a handful of characters and it adds value for several situations, so I'm willing to try it out
  • updated the documentation and unit tests regarding this
  • the client api version is now 54

  • boring stuff

  • file filter objects are now serialisable
  • application commands can now hold serialisable objects in their 'simple data' slot
  • I made a new 'slightly more than simple' application command to hold a 'thumbnail move' that has both a direction and a selection status. I expect it will be expanded in future to handle ctrl+ selection and other logic preferences
  • I made a new application command to hold the file filter. I just pre-populate the UI with a dropdown with commond choices for now, but in future it could hold a customisable file filter, once, ha ha, I have some UI to actually edit one!
  • cleaned up various shortcut code
  • misc linting cleanup

• version 547

  • mpv crash fixes

  • tl;dr: mpv less crashy now
  • if mpv fails to load a file but not in an outright 'error' manner (this appears to mean a file using a rare format that a submodule of mpv can't handle), the client now recognises this has happened, either right after the first load, or, if the error takes longer to occur, a subsequent status interrogation, and makes several new steps to restore program stability: disconnecting the mpv window from all commands, freezing the scanbar, loading the default hydrus.png as emergency backstop, and making a popup to let the user know what just happened. previously, Qt would get rapidly unhappy as it asked things to draw on screen over the null-state player, particularly if you show/hid the scanbar several times, and it would, if not removed promptly from screen, typically lead to a program crash
  • furthermore, the scanbar now never interrogates the mpv window during its paint event. a mysterious interaction of C++ level objects during error state was causing the underlying instability here, and now I cannot reproduce this even if I try
  • I also hardened the mpv window's 'no-media' state. now, rather than showing 'nothing' when media is unloaded, each mpv player now actually idles on a black png lol
  • this tech will kick in for more extreme file failures, too, which have a different handler but seem to give the same detectable dump-out state
  • fixed a silent-but-for-debug-mode error while destroying damaged mpv windows right when the program is terminating

  • misc

  • thanks to a user, we now have import support for 'djvu' files. basically an open source PDF style format
  • fixed pasting an image into 'system:similar files', which I missed updating in last week's code cleanup!
  • a light but spammy legacy job that refreshed every search page's empty autocomplete every five minutes (to get updated system predicates/numbers) now only occurs to autocompletes on the current page. relatedly, when you switch to a search page you haven't looked at in five minutes, it triggers the same update immediately. this should save a tiny bit of idle CPU time and, more importantly, clear out the background job queue on larger-session clients
  • I _think_ I fixed some instances of the media viewer notes window initialising with a gigantic width on some OSes. if you often get a super wide notes window when you first open the media viewer, with it fixing itself when you cycle to a different file and back, let me know if things are any better
  • when you have a popup message that has a 'show x files' button, usually from a subscription, that routine now excludes files that have been deleted since the button was created. it updates its existing file count on a click, also, to how many files it actually will generate. if you click one of these buttons, delete some files, and then click it again, it should no longer produce ghost files in the new search page. I'm going to add some more tech to optionally handle the system:hash predicate in a page in similar ways, 'locking' it to the current page content and preserving file sort so it works nice with 'remove files' etc..
  • fixed a stupid typo that was swapping the 'allow non-local connections' server setting when making the interface for IPv6 hosts. there is a secondary check of all client IPs on every request, so I am confident this was not enabling non-local connections when undesired on IPv6, but it was disabling them by deploying the loopback interface when they should have been allowed! sorry for the trouble, and well done to the person who noticed this
  • while pursing an odd and rare problem where a download job can start even though it should be waiting on a login process, I cleaned some of the login code and logic, lowering the timeout for session cookie expiring from 60 to 45 minutes and smoothing out some confusing status-checking in the pre-login stage. I could never reproduce the problem, though, so if you have had this issue, please let me know more and I'll see if I can reproduce this reliably

  • simple cleanup

  • cleaned up some filetype parsing code that was getting a little messy, also reduced some overhead
  • unified the thumbnail/file filetype parsing a little, with better fallback states when a hydrus thumbnail happens for some reason not to be a jpeg or png
  • fixed an out of date menu reference in the 'help my media files are broke.txt' document. 'clear orphan files' is under 'file maintenance' now, not 'db maintenance'

• version 546

  • misc

  • fixed the recent messed up colours in PSD thumbnail generation. I enthusiastically 'fixed' a problem with greyscale PSD thumbs at the last minute last week and accidentally swapped the RGB colour channels on coloured ones. I changed the badly named method that caused this mixup, and all existing PSD thumbs will be regenerated (issue #1448)
  • fixed up some borked button-enabling and status-displaying logic in the file history chart. the cancel process should work properly on repeat now
  • made two logical fixes to the archive count in the new file history chart when you have a specific search--archive times for files you deleted are now included properly, and files that are not eligible for archiving are discluded from the initial count. this _should_ make the inbox and archive lines, which were often way too high during specific searches, a little better behaved. let me know what you see!
  • added a checkbox to _options->thumbnails_ to turn off the new blurhash thumbnail fallback
  • 'this has exif data, the other does not' statements are now calculated from cached knowledge--loading pairs in the duplicate filter should be faster now
  • some larger image files with clever metadata should import just a little faster now
  • if the process isn't explicitly frozen into an executable or a macOS App, it is now considered 'running from source'. various unusual 'running from source' modes (e.g. booting from various scripts that mess with argv) should now be recognised better

  • boring code cleanup

  • moved 'recent tags' code to a new client db module
  • moved ratings code to a new client db module
  • moved some db integrity checking code to the db maintenance module
  • moved the orphan table checking code to the db maintenance module
  • fixed the orphan table checking code, which was under-detecting orphan tables
  • moved some final references to sibling/parent tables from main db method to sibling and parent modules
  • moved most of the image metadata functions (exif, icc profile, human-readable, subsampling, quantization quality estimate) to a new `HydrusImageMetadata` file
  • moved the new blurhash methods to a new `HydrusBlurhash` file
  • moved various normalisation routines to a new `HydrusImageNormalisation` file
  • moved various channel scanning and adjusting code to a new `HydrusImageColours` file
  • moved the hydrus image files to the new 'hydrus.core.files.images' module
  • cleaned up some image loading code
  • deleted ancient and no-longer-used client db code regarding imageboard definitions, status texts, and more
  • removed the ancient `OPENCV_OK` fallback code, which was only used, superfluously, in a couple of final places. OpenCV is not optional to run hydrus, server or client

• version 545

  • blurhash

  • thanks to a user's work, hydrus now calculates the [blurhash](https://blurha.sh/) of files with a thumbnail! (issue #394)
  • if a file has no thumbnail but does have a blurhash (e.g. missing files, or files you previously deleted and are looking at in a clever view), it now presents a thumbnail generated from that blurhash
  • all existing thumbnail-having files are scheduled for a blurhash calculation (this is a new job in the file maintenance system). if you have hundreds of thousands of files, expect it to take a couple of weeks/months to clear. if you need to hurry this along, the queue is under _database->file maintenance_
  • any time a file's thumbnail changes, the blurhash is scheduled for a regen
  • for this first version, the blurhash is very small and simple, either 15 or 16 cells for ~34 bytes. if we end up using it a lot somewhere, I'd be open to making a size setting so you can set 8x8 or higher grids for actually decent blur-thumbs
  • a new _help->debug_ report mode switches to blurhashes instead of normal thumbs

  • file history search

  • I did to the file history chart (_help->view file history_) what I did to mr bones a couple weeks ago. you can now search your history of imports, archives, and deletes for creator x, filetype y, or any other search you can think of
  • I hacked this all together right at the end of my week, so please bear with me if there are bugs or dumb permitted domains/results. the default action when you first open it up should all work the same way as before, no worries™, but let me know how you get on and I'll fix it!
  • there's more to do here. we'll want a hideable search panel, a widget to control the resolution of the chart (currently fixed at 7680 to look good blown up on a 4k), and it'd be nice to have a selectable date range
  • in the longer term future, it'd be nice to have more lines of data and that chart tech you see on financial sites where it shows you the current value where your mouse is

  • client api

  • the `file_metadata` call now says the new blurhash. if you pipe it into a blurhash library and blow it up to an appopriate ratio canvas, it _should_ just work. the typical use is as a placeholder while you wait for thumbs/files to download
  • a new `include_blurhash` parameter will include the blurhash when `only_return_basic_information` is true
  • `file_metadata` also shows the file's `pixel_hash` now. the algorithm here is proprietary to hydrus, but you can throw it into 'system:similar files' to find pixel dupes. I expect to add perceptual hashes too
  • the help is updated to talk about this
  • I updated the unit tests to deal with this
  • the error when the api fails to parse the client api header is now a properly handled 400 (previously it was falling to the 500 backstop)
  • the client api version is now 53

  • misc

  • I'm sorry to say I'm removing the Deviant Art artist search and login script for all new users, since they are both broken. DA have been killing their nice old API in pieces, and they finally took down the old artist gallery fetch. :(. there may be a way to finagle-parse their new phone-friendly, live-loading, cloud-deployed engine, but when I look at it, it seems like a much bigger mess than hydrus's parsing system can happily handle atm. the 'correct' way to programatically parse DA is through their new OAuth API, which we simply do not support. individual page URLs seem to still work, but I expect them to go soon too. Sorry folks, try gallery-dl for now--they have a robust OAuth solution
  • thanks to a user, we now have 'epub' ebook support! no 'num_words' support yet, but it looks like epubs are really just zips with some weird metadata files and a bunch of html inside, so I think this'll be doable with a future hacky parser. all your existing zip files wil be scheduled for a metadata rescan to see if they are actually epubs (this'll capture any secret kritas and procreates, too, I think)
  • the main UI-level media object is now aware of a file's pixel hash. this is now used in the duplicate filter's 'these are pixel duplicates' statements to save CPU. the jank old on-the-fly calculation code is all removed now, and if these values are missing from the media object, a message will now be shown saying the pixel dupe status could not be determined. we have had multiple rounds of regen over the past year and thus almost all clients have full database data here, so fingers crossed we won't see this error state much if at all, but let me know if you do and I'll figure out a button to accelerate the fix
  • the thumbnail _right-click->open->similar files_ menu now has an entry for 'open the selection in a new duplicate filter page', letting you quickly resolve the duplicates that involve the selected files
  • pixel hash and blurhash are now listed, with the actual hash value, in the _share->copy->hash_ thumbnail right-click menu
  • thanks to a user, 'MPO' jpegs (some weird multi-picture jpeg that we can't page through yet) now parse their EXIF correctly and should rotate on a metadata-reparse. since these are rare, I'm not going to schedule a rescan over everyone's jpegs, but if you see a jpeg that is rotated wrong, try hitting _manage->regenerate->file metadata_ on its thumbnail menu
  • I may have fixed a rare hang when highlighting a downloader/watcher during very busy network time that involves that includes that importer
  • added a warning to the 'getting started with installing' and 'database migration' help about running the SQLite database off a compressed filesystem--don't do it!
  • fixed thumbnail generation for greyspace PSDs (and perhaps some others)

  • boring cleanup

  • I cleaned some code and added some tests around the new blurhash tech and thumbs in general
  • a variety of metadata changes such as 'has exif', 'has icc profile' now trigger a live update on thumbnails currently loaded into the UI
  • cleaned up some old file metadata loading code
  • re-sorted the job list dropdown in the file maintenance dialog
  • some file maintenance database work should be a bit faster
  • fixed some behind the scenes stuff when the file history chart has no file info to show

• version 544

  • webp vulnerability

  • the main webp library (libwebp) that many programs use for webp support had a remote execution (very bad) vulnerability. you probably noticed your chrome/firefox updated this week, which was fixing this. we use the same thing via the `Pillow` library, which also rolled out a fix. I'm not sure how vulnerable hydrus ever was, since we are usually jank about how we do anything, but best to be safe about these things. there were apparently exploits for this floating around
  • the builds today have the fix, so if you use them, update as normal and you are good
  • if you run from source, **rebuild your venv at your earliest convenience**, and you'll get the new version of Pillow and be good. note, if you use the advanced setup, that there is a new question about `Pillow`
  • unfortunately, Windows 7 users (or anyone else running from source on Python 3.7) cannot get the fix! it needs Pillow 10.0.1, which is >=Python 3.8. it seems many large programs are dropping support for Win 7 this year, so while I will continue to support it for a reasonable while longer, I think the train may be running out of track bros

  • max size in file storage system

  • the `migrate database` dialog now allows you to set a 'max size' for all but one of your media locations. if you have a 500GB drive you want to store some stuff on, you no longer have to balance the weights in your head--just set a max size of 450GB and hydrus will figure it out for you. it is not super precise (and it isn't healthy to fill drives up to 98% anyway), so make sure you leave some padding
  • also, please note that this will not automatically rebalance _yet_. right now, the only way files move between locations is through the 'move files now' button on the dialog, so if you have a location that is full up according to its max size rule and then spend a month importing many files, it will go over its limit until and unless you revisit 'migrate database' and move files again. I hope to have automatic background rebalancing in the near future
  • updated the 'database migration' help to talk about this and added a new migration example
  • the 'edit num bytes' widget now supports terabytes (TB)
  • I fleshed out the logic and fixed several bugs in the migration code, mostly to do with the new max size stuff and distributing weights appropriately in various situations

  • misc

  • when an image file fails to render in the media viewer, it now draws a bordered box with a brief 'failed to render' note. previously, it janked with a second of lag, made some popups, and left the display on eternal blank hang. now it finishes its job cleanly and returns a 'nah m8' 'image' result
  • I reworked the Mr Bones layout a bit. the search is now on the left, and the rows of the main count table are separated for readability
  • it turns out that bitmap (.bmp) files can support ICC Profiles, so I've told hydrus to look for them in new bitmaps and retroactively scan all your existing ones
  • fixed an issue with the recent PSD code updates that was breaking boot for clients running from source without the psd-tools library (this affected the Docker build)
  • updated all the 'setup_venv' scripts. all the formatting and text has had a pass, and there is now a question on (n)ew or (old) Pillow
  • to stop FFMPEG's false positives where it can think a txt file is an mpeg, the main hydrus filetype scanning routine will no longer send files with common text extensions to ffmpeg. if you do have an mp3 called music.txt, rename it before import!
  • thanks to a user, the inkbunny file page parser fetches the correct source time again (#1431)
  • thanks to a user, the old sankaku gallery parser can find the 'next page' again
  • removed the broken sankaku login script for new users. I recommend people move to Hydrus Companion for all tricky login situations (#1435)
  • thanks to a user, procreate file parsing, which had the width/height flipped, is fixed. all existing procreate files will regen their metadata and thumbs

  • client api

  • thanks to a user, the Client API now has a `/get_files/render` command, which gives you a 100% zoom png render of the given file. useful if you want to display a PSD on a web page!
  • I screwed up Mr Bones's Client API request last week. this is now fixed
  • Mr Bones now supports a full file search context on the Client API, just like the main UI now. same parameters as `/get_files/search_files`, the help talks about it. He also cancels his work early if the request is terminated
  • Mr Bones gets several new unit tests to guarantee long-term ride reliability
  • the Client API (and all hydrus servers) now return proper JSON on an error. there's the error summary, specific exception name, and http status code. the big bad 500-error-of-last-resort still tacks on the large serverside traceback to the summary, so we'll see if that is still annoying and split it off if needed
  • the new `/add_tags/get_siblings_and_parents` now properly cleans the tags you give it, trimming whitespace and lowercasing letters and so on
  • the client api version is now 52

• version 543

  • misc

  • a new string converter rule now allows for extremely easy date parsing, thanks to the `dateparser` library. all old 'datestring to timestamp' rules remain as they are, but are now called '(advanced)'. a new option, 'datestring to timestamp (easy)', which has exactly zero variables to fiddle with, just eats up pretty much any date string you can think of, including timezone conversions, and even stuff like '2 hours ago'. you need the dateparser library for this to work, so **if you run from source, you might like to rebuild your venv this week**. your `dateparser` import status is in _help->about_
  • thanks to the user who added it recently, PSD rendering is now much faster and uses less memory. if you do a lot of PSD work, let me know how this goes. if PSDs now load pretty much like large pngs, I think we'll set them, by default, to show as normal in the preview viewer
  • thanks to a user, we now have description note parsing for the default e621 downloader
  • the program now supports bitmap files as-is. until now, I automatically converted them to png on import, but this was a mistake--despite this file format being a waste 99.7% of the time, hydrus's philosophy is not to alter files on import, and this long-time exception resulted in several awkward bumps in the code that I'm happy to be rid of now
  • fixed a couple desync bugs in the migrate database dialog where you could change a location's weight (particularly between 0 and 1) and not get the correct flip of the 'files need to be moved'/'files are all good' state until you re-opened the dialog

  • PDFs

  • I screwed something up with the PDF thumbnail generation at the last minute last week, fixing it on non-PySide6, but introducing some logspam and--for at least one user--adding instability. the logspam is now gone and I _believe_ the instability is fixed. now it is basically the same as the SVG thumbnail code, which hasn't given us any trouble. if we still see some crashes, I'm going to have to overhaul these two thumbnail generation methods
  • when PDFs fail to generate thumbs, a little text about the error is now printed to the log
  • _help->about_ now has lines for QtCharts and QtPdf, and if there is a PDF problem, it puts the import trace in a popup

  • mr bones

  • mr bones can now take any file search. if you want to see the average filesize of your pngs, or the archive/inbox ratio of creator x's webms, just set that search on the new panel and the numbers will update for that subset
  • this turned, characteristically, into a bottomless rabbit hole, and I culled the more complicated features lest the ride consume me. searching a multiple file domain means deleted numbers cannot be calculated, nor can the 'earliest import' time, and searching deleted domains will generally give you some gonk numbers (and likely reveal some interesting legacy bugs, like inbox count amongst deleted files)
  • the old search was highly optimised, but this has few guard rails. if you give this thing a super difficult query, it'll take a long time. there is now a cancel button that should interrupt all but the weirdest operations fairly promptly, however, just in case it is really lagging. note that hitting 'searching immediately' will pause updates as normal, if you need to set up something complex
  • assuming deleted numbers are available, the stats now include total views/viewtime for deleted files too
  • potential dupe counts are basically a search of 'at least one of the files matches the file list, can be pixel dupes, max distance 8'

  • more boring work, file storage and misc

  • wrote a new object to handle the base storage location for file/thumbnail subfolders. it can do over/underweight calculations and handles the pending max_num_bytes setting for database migration locations
  • all the new subfolder objects now track their base location using this new object, and all related load/save/display/edit code is now throwing this thing around instead of raw paths
  • the underlying migration determination code is now ready to redistribute according to a max_num_bytes option. I've just got to update the UI, and, fingers crossed, I'll be able to add it next week
  • added a bunch of unit tests for the new base storage location object. it separately reports whether it needs to shrink, wants to shrink, is able to expand, or is eager to expand
  • improved how updated objects are substituted into all multi-column lists, it fixes a couple of odd storage/display sync bugs here and there
  • a core image data loading/conversion tool inside the program is now a bit simpler and faster, and I think it also saves memory. it should speed up various sorts of unusual file loading

• version 542

  • pdfs

  • thanks to a user, we now have pdf thumbnails! there is surprisingly little jank!
  • I hacked together a newer and better word count for PDFs. I can't promise it is perfect, but it does actually inspect the raw text. I'm expect we'll add a separate 'num_pages' row in future to handle comics (and other stuff like cbr/cbz)
  • I also hacked in 'human-readable file metadata' for PDFs. any PDF with author, title, subject, or keywords metadata is now viewable at the top of the media viewer
  • on update all your existing pdfs will be scheduled to get new thumbs, count their words, and learn if they have human-readable file metadata
  • this tech relies on Qt, so users running from source on old OSes (and thus Qt5) may not have very good support, sorry!

  • predicate parsing

  • the system predicate parser can now deal with numbers with commas, like in `system:width = 1,920`
  • `system:filetype is gif` works again in the predicate parser, now resolving to `system:filetype is animated gif, static gif`
  • fixed some weird parsing for 'system:tag as number' and added more operators like 'less than' and support for 'unnamespaced' and 'any namespace'
  • `system:tag as number` now labels itself in the client in the style `system:tag as number: page less than 20`, which is parseable by the system
  • the predicates for 'has exif/icc profile/human-readable embedded metadata' now label themselves in the format `system:has x`, not `system:image has x`. this harmonises with our other `has x` predicates, recognises that we pull metadata from non-images these days, and is the text that they were parsing with anyway

  • misc

  • the 'exporting' sidecar system's 'tag' source (i.e. pulling tags from your local tag services) now has a button to select 'storage' (no siblings or parents, what you see in manage tags dialog) or 'display' (has sibling and parent calculations, what you see in normal views) tags. all existing tag source sidecars will stay 'storage', but the default for new ones is now 'display'
  • renamed the dumb 'x metadata migrations' button label in export files to 'x sidecar actions'
  • wrote a new FAQ answer about why tags don't disappear when you delete files: https://hydrusnetwork.github.io/hydrus/faq.html#service_isolation
  • also wrote just a little FAQ about running hydrus off an encrypted partition--yes you can, and this is good tech to learn
  • moved the builds up to python 3.10. I thought we had already done this, but there we go. no special install instructions, it should just update as normal
  • for users who run from source: added a '(m)iddle Qt6' selection to the advanced setup venv script, for those who cannot run 6.5.2, with some explanation about it (it is the recently used 6.4.1, since Python 3.11 can't run the '(o)lder' 6.3.1), and added a '(t)est mpv' option for the newer python-mpv 1.0.4

  • boring file storage work

  • I decided that the planned granular folders will nest in groups of 2 hex characters. when you move to three-character storage, the files starting 'ab1' will be stored in '/fab/1' directory (rather than '/fab1'). we don't want to solve the overhead of a folder with 30,000 files by creating a folder with 4096 or 65536 folders. all the code was shifted over to this
  • all the migrate and repair code now uses subfolders
  • replaced various hardcoded folder determination code with subfolders, ensuring we are all calculating locations using the same single method
  • a variety of other responsibilities like 'does this subfolder exist on disk?' and 'make sure it does exist' are similarly now all collected in one place, in the subfolder code
  • added a little suite of unit tests for the new subfolders class
  • did a bunch of renaming to clear up various different concepts and names in all this code
  • the 'clear' custom thumbnail location button in migrate database is now wrapped in a yes/no confirmation dialog

  • other boring stuff

  • wrote some new exception classes to handle several 'limited support for this particular file' states and refactored a bunch of the resolution and thumbnail producing code to use it instead of None hacks or 'this file is completely busted' exceptions
  • improved some misc file format handling, particularly when they are damaged. stuff like clip database inspection and general thumbnail generation fail states
  • refactored many of my hardcoded special unicode characters to constants in HC. not sure I really like all the spammed `{HC.UNICODE_ELLIPSIS}` though, so might revisit
  • fixed an issue with last week's update code that was affecting users with a history of certain database damage
  • I may have improved import support for some damaged or generally very strange image files by falling back to OpenCV for resolution parsing when Pillow fails

• version 541

  • misc

  • fixed the gallery downloader and thread watcher loading with the 'clear highlight' button enabled despite there being nothing currently highlighted
  • to fix the darkmode tooltips on the new Qt 6.5.2 on Windows (the text is stuck on a dark grey, which is unreadable in darkmodes), all the default darkmode styles now have an 'alternate-tooltip-colour' variant, which swaps out the tooltip background colour for the much brighter normal widget text colour
  • rewrote the apng parser to work much faster on large files. someone encountered a 200MB giga apng that locked up the client for minutes. now it takes a second or two (unfortunately it looks like that huge apng breaks mpv, but there we go)
  • the 'media' options page has two new checkboxes--'hide uninteresting import/modified times'--which allow you to turn off the media viewer behaivour where import and modified times similar to the 'added to my files xxx days ago' are hidden
  • reworked the layout of the 'media' options page. everything is in sections now and re-ordered a bit
  • the 'other file is a pixel-for-pixel duplicate png!' statements will now only show if the complement is a jpeg, gif, or webp. this statement isn't so appropriate for formats like PSD
  • a variety of tricky tags like `:>=` are now searchable in normal autocomplete lookup. a test that determined whether to use a slower but more capable search was misfiring
  • the client api key editing window has a new 'check all permissions' button
  • fixed the updates I made last week to the missing-master-file-id recovery system. I made a stupid typo and didn't test it properly, fixed now. sorry for the trouble!
  • thanks to a user, the help has a bunch of updated screenshots and fixed references to old concepts
  • did a little more reformatting and cleanup of 'getting started with downloading' help document and added a short section on note import options
  • cleaned up some of the syntax in our various batch files. fingers crossed, the setup_venv.bat script will absolutely retain the trailing space after its questions now, no matter what whitespace my IDE and github want to trim

  • string joiner

  • the parsing system has a new String Processor object--the 'String Joiner'. this is a simple concatenator that takes the list of strings and joins them together. it has two variables: what joining text to use, e.g. ', ', or '-', or empty string '' for simple concatenation; and an optional 'group size', which lets you join every two or three or n strings in 1-2-3, 1-2-3, 1-2-3 style patterns

  • new file types

  • thanks to a user; we now have support for QOI (a png-like lossless image type) and procreate (Apple image project file) files. the former has full support; the latter has thumbnails
  • QOI needs Pillow 9.5 at least, so if you are on a super old 'running from source' version, try rebuilding your venv; or cope with you QOI-lessness

  • client api

  • thanks to a user, we now have `/add_tags/get_siblings_and_parents`, which, given a set of tags, shows their sibling and parent display rules for each service
  • I wrote some help and unit tests for this
  • client api version is now 51

  • file storage (mostly boring)

  • the file storage system is creaky and ugly to use. I have prepped some longer-term upgrades, mostly by writing new tools and cleaning and reworking existing code. I am nowhere near to done, but I'd like us to have four new features in the nearish future:
  • - dynamic-length subfolders (where instead of a fixed set of 256 x00-xff folders, we can bump up to 4096 x000-xfff, and beyond, based on total number of files)
  • - setting fixed space limits on particular database locations (e.g. 'no more than 200GB of files here') to complement the current weight system
  • - permitting multiple valid locations for a particular subfolder prefix
  • - slow per-file background migration between valid subfolders, rather than the giganto folder-atomic program-blocking 'move files now' button in database maintenance
  • so, it is pretty boring so far, but I did the following:
  • wrote a new class to handle a specific file storage subfolder and spammed it everywhere, replacing previous location and prefix juggling
  • wrote some new tools to scan and check the coverage of multiple locations and dynamic-length subfolders
  • rewrote the file location database initialisation, storage, testing, updating, and repair to support multiple valid locations
  • updated the database to hold 'max num bytes' per file storage location
  • the feature to migrate the SQLite database files and then restart is removed from the 'migrate database' dialog. it was always ultrajank in a place that really shouldn't be, and it was completely user-unfriendly. just move things manually, while the client is closed
  • the old 'recover and merge surplus database locations into the correct position' side feature in 'move files now' is removed. it was always a little jank, was very rarely actually helpful, and had zero reporting. it will return in the new system as a better one-shot maintenance job
  • touched up the migrated database help a little

• version 540

  • misc

  • the system predicate parser can now handle 'system:filetype is xxx' for more of the general human-friendly filetype strings like 'video' and 'mkv'. it can also handle 'static gif' and any other types with spaces but now enforces commas between each filetype. I think all system:filetype predicate strings the client produces now parse correctly if you paste them back
  • fixed many bitmap imports, most typically in the 'system:similar files' system, which was not generating pixel hashes correctly. most/all bitmaps coming in with alpha channels, or, I also believe, with a null channel (RGB32), were being handled wrong and coming out BGR. perceptual hashes are greyscale and were not affected, but pixel hashes were wrong. this was a real pain to figure out, and it may be that it is still broken for users on big-endian systems or something, so let me know how you get on
  • added links to https://github.com/abtalerico/wd-hydrus-tagger (danbooru-trained model tagging) and https://github.com/Garbevoir/wd-e621-hydrus-tagger (which adds more models) to the client api help. reports are that they work well, even on 'normal' pictures
  • the bad darkmode tooltip text colour in the new Qt 6.5.2 on Windows appears to be a bug, here: https://bugreports.qt.io/browse/QTBUG-116021 . there's not a great answer here, so let me know your thoughts. if you like, you can edit a custom stylesheet with a different `QToolTip` `background-color`, or I can spam some alternate fixed QSS files for everyone, or we can wait for a fix on Qt's end
  • on update, all existing PSD and static gif files will be scheduled for pixel hash regen, perceptual hash regen, and entry into the similar files system (I forgot to do this last week)
  • on update, all existing PNGs will be scheduled for pixel duplicate data regen. we have a legacy alpha channel issue here that has reared its head several times (searchting for 'must not be pixel dupes', but getting pixel dupes), so I am just going to bosh it on the head for everyone

  • file maintenance

  • if a file has multiple jobs pending, the file maintenance manager now processes all those jobs at once, saving significant disk I/O. also, a couple things like the 'do all work' button's popup now shows the total number of jobs to do, rather than that of each job type in turn
  • the 'manage scheduled jobs' file maintenance panel now shows the count for jobs that exist but are not yet due. previously, these were hidden, which was part of the mkv/webm duplicate difficulties last week.
  • when the program needs to rename a file because it has a new mime, it now first tests if the file is still in use (normally this means some file parsing component like ffmpeg or opencv is still cleaning up OS file handles or whatever), and, if so, waits just a little bit before trying
  • relatedly, the 'try and delete the rename-dupe again' job now tries again in one hour, rather than one week in the future, and if that after-one-hour job fails again (this would usually be because you were actually viewing the original file in the media viewer at the time of its reparse), then that job will retry again in a week, and the week after if that fails, and again after that, etc... for about a year
  • fixed an issue with the thumbnail resizing maintenance job on PSD files and probably some other weird types too
  • fixed some scheduling issues in how the mainloop of the file maintenance system tests its current rate of work and when it should cancel a current batch of work

  • boring stuff

  • simplified and cleaned up some of the duplicate system king-fetching code. I _may_ have also fixed one instance of pair representatives being fetched wrong for the filter when 'at least one file has to match the one search'
  • when editing the duplicate action merge options, a new label at the top says which dupe action you are editing for, and if it isn't "this is better", it notes that the available merge actions are limited
  • improved four things with the recovery code that handles missing master hash definitions--first, the substitute hashes are now the correct length; second, they are now saved back to the database, which should stop issues like the "trying to delete a thing that doesn't exist and has an ever-changing name in a loop forever" bug; third, the popup tells the user what to do next, and more information is written to the log; and fourth, the client checks the local hash cache so see if it can automatically recover the missing data

  • client api

  • the `file_metadata` call now has two new fields, `filetype_human`, which looks like 'jpeg' or 'webm', and `filetype_enum`, which uses internal hydrus filetype numbers
  • the help and unit tests are updated for this
  • client api version is now 50

• version 539

  • another new library today

  • **if you run from source, I recommend you rebuild your venv today. we've got another library to add full PSD support**

  • viewable PSD files

  • thanks to a user, hydrus can now view PSD files in the main viewer, just like any other file. it can be a bit slow to load, and you can't show/hide away layers or anything, but for simply showing the image as-is, it works great
  • because this needs more CPU than for normal images, we're starting out with conservative view settings. while PSD files will show as normal in the media viewer, they'll only show the 'open externally' button in the preview viewer. see how it works for you, and remember you can change it under _options->media_. if there isn't any real problem IRL with showing even big PSDs everywhere, I'll change the defaults, so let's see how it goes
  • and of course if you have a borked PSD--say one that shows up in all the wrong colours, or has bad transparency--please send it in and I'll have a look. there is apparently a rare class of PSD files that simply won't render at all with our new system, and hydrus is pretty bad at handling that situation, so that'd also be useful if at the very least to get me to write some better error handling code
  • just like last week, if you run from source, please rebuild your venv again today--there's a PSD-handling library that supports all this
  • all PSD files will be scheduled for a thumbnail regen on update

  • static vs animated gif

  • the program now treats still and animated gifs as separate file types for the purpose of searching and selecting display options (previously, static gifs were just animated gifs with no duration). most people won't have many static gifs, so this doesn't matter too much, but it cleans up our image/animation filetype group distinction and makes a bunch of behind the scenes stuff simpler. all your gifs will be set to either camp on update
  • if you have an existing file import options or system:filetype that looked for gifs specifically, it will now search for 'animated gifs' only, so watch out if you need 'static gif' too/instead

  • parseable rating predicates

  • the system predicate parser (including the Client API) now accepts system:rating predicates. type 'system:has rating (service name)' or 'system:rating for (service name) > 4/5' or other reasonable variants and it should pre-fill
  • in the UI, the system 'has/no rating' predicate strings are now in the format 'has a rating for (service name)' and 'does not have a rating for (service name)'. (previously it was 'has/no (service name) rating', which is out of step with our usual syntax and generally unhelpfully parsing-ambiguous)
  • added a bunch of unit tests for this

  • misc

  • fixed the 'network timeout' setting under _options->connection_, which was not saving changes
  • the media viewer top hover window now enables/disables the 'show file metadata' button--rather than shows/hides--in order to stop the buttons on the left jumping around so much when you scroll through media
  • the duplicate filter's always-on-top right-hand window is fixed in place again. the buttons won't jump around any more. if the notes hover grows to overlap it, it won't show over it as long as your mouse is over the duplicate hover. this should make clicking those duplicate buttons on note-heavy files far less frustrating. sorry for the late fix here!
  • the duplicate filter now always presents a statement on the pair's filetypes, even if they are the same (it'll say like 'both are pngs'). this is to help catch the upcoming PSD matches (where you probably do not want to delete either) and other weirdness as we add new filetypes
  • just a small thing, but the 'management panel' labels are renamed broadly to 'sidebar' under the _pages_ menu. the panel on the left of pages is now called 'sidebar', and the wx-era 'sash' wording is gone
  • there's an issue with the file metadata reparse system right now where, on a filetype change, it will often fail to cleanly rename bigger files (e.g. from x.mkv to x.webm). the result is the file copies and the old one is never deleted, leaving a duplicate that is not properly cleared up later. on update this week, I am scheduling a fresh cleanup for these dupes. if, like me, you have a lot of large AV1-encoded vidya capture mkvs, you may have noticed your hydrus folders suddenly bloated in size this past week--this should be 99% fixed soon. I will fix the underlying issue here next week
  • touched up the 'running from source' help and 'setup_venv' texts in general and specifically regarding some new version info stuff. it looks like macOS <= 10.15 can't handle last week's new Qt6 version, and some versions of Linux need `libicu-dev` and `libxcb-cursor-dev` installed via apt or otherwise
  • fixed the file query sort-and-clip method when you are set to sort by file hash and also have system:limit, and fixed it for asc/desc too
  • for the second time, fixed the 'import QtSVG' error on hydrus server install when the client requirements.txt had not been run. turns out I messed up the 'proper' fix I did for the first time

  • boring cleanup

  • refactored a bunch of HydrusImageHandling code to HydrusAnimationHandling
  • cleaned up several of our enums and enum testing, and cleared out several hardcoded hooks to deal with different kinds of gif
  • did some similar enum cleaning and `gif`->`PILAnimation` renaming to encompass the new HEIF sequences
  • streamlined the image and animation metadata parsing methods significantly
  • a bunch of simple `image`->`animation` renames, like IMAGE_APNG is now ANIMATION_APNG
  • cleaned up some other confusing code handles for 'image' vs 'static image', to handle whether we are talking about strictly images or viewable raster image-likes (for now including PSD files) but I think it'll need more work
  • deleted some ancient and no longer used imageboard profile code

• version 538

  • important note on index regeneration

  • **if you get a note on update about missing indices that need to be regenerated, don't panic! everything is fine, nothing to worry about, let it do its work**

  • new libraries today

  • **if you run from source, I recommend you rebuild your venv today. the setup script points at new versions of Qt, OpenCV, and a HEIF module that adds new filetype support**

  • new Qt and OpenCV

  • all release builds and normal source installations move up to PySide6 (Qt) 6.5.2 today. we've done a good bit of testing in different situations, and it seems to be a good and reliable upgrade from 6.4.1, which has given us a mix of annoying trouble at times, like mismatched UI scaling and mpv-related flickering
  • let me know if you have any trouble with the overall feel of the program, particularly if you are running on an older or un-updated version of your OS
  • the last version of Qt that was generally without caveats was 6.3.1. if you do have trouble with today's release (I suspect old and un-updated OSes, or source users on older Python), one option is to move to running from source and using this older version, which I have updated my setup_venv scripts to offer as a stable 'Qt6 (o)lder' option
  • similarly, we are moving from OpenCV (an image library) 4.5.5.64 to 4.7.0.72. we've tested this in several rounds of future-builds and had no reports of trouble, and this also improves some build compatibility with FFMPEG 5.0 (issue #1419)
  • the 'test' version of Qt stays at 6.5.2 for now, since this is the latest version
  • the 'old' OpenCV compatability version remains at 4.5.3.56, the new 'test' version is now 4.8.0.74

  • deferred delete system

  • the first full version of the deferred delete system is complete. your no-longer-needed tables lying around after a big operation like a PTR delete/reset will now be shrunk in the background until they are small enough to delete in trivial time
  • the menu entry under _database->database maintenance_ has a new submenu for the job and 'work in idle/normal' time checkboxes just like file maintenance
  • the new review window UI is now fleshed out. it can refresh itself, and automitically does so on changes, and the 'work hard' button functions
  • I discovered a bug in last week's code that stopped some indices from being recreated in certain regeneration jobs. if you did a 'regenerate tag text search cache' or similar operation last week, you'll encounter the above 'need to regen some indices' note. no worries, it'll all fix itself, and, if you noticed any slowdown, the affected system should work at the proper speed again

  • user contributions

  • thanks to a user, we now have full support for HEIF, HEIC, and AVIF image files. they will import and render just like any other image. furthermore, we have support for HEIF, HEIC, and AVIF 'sequences', which are basically like an animated gif or apng and are under the 'animations' filetype category (and they seem to play in mpv great! although I don't have an example HEIC sequence to test with, lol). all users who use the normal build will get this on update--anyone running from source will want to rebuild their venv this week to get the functionality. you can double-check _help->about_ to see if you have the required 'pillow-heif' library
  • thanks to a user, the various help links in the program now redirect to the online help (and/or direct to a guide to build the local help) if the local help is missing (fixes #1360)
  • thanks to a user, an addititonal final network transfer size check is now in place. if a server says it will deliver x bytes and actually delivers y, the job now raises an error. this can happen with various twitter solutions, where vid downloads will sometimes stealth-stop-working, leaving a valid but truncated mp4. fingers crossed this will now catch that situation and trigger a re-attempt
  • thanks to a user, fixed TIFF files not showing EXIF correctly. just to be safe, all tiffs will be scheduled for a 'has EXIF?' rescan on update, and I silenced another bit of tiff-related PIL warning-logspam
  • mkv files with AV1 video (and no/worbis/opus audio) are now correctly identified as webms. all mkvs will be scheduled for a metadata rescan on update

  • misc

  • when transferring mappings, _tags->migrate tags_ now supports a full location context file filter (like the file domain button you see in an autocomplete). previously it was just a list of single locations to pick from, but now, if you want to grab tags for all files deleted from x, or all files in either y or z, it is simple to set up. relatedly, the 'multiple/deleted' dialog picker launched from that menu now sizes itself to try and fit all its stuff in, rather than always being scrolled
  • fixed a bug that meant you could ok the 'edit predicate' dialog despite the regex string being invalid when editing an existing 'system:known url=(regex)' predicate. the 'check valid' test is now caught correctly and cancels dialog ok, rather than escalating to the popup message catcher
  • fixed a bug in table analyze code that was causing empty tables to be unintentionally re-analysed over good existing data
  • a file_system_type-checking call that is used in file-export is now cached. previously it was hit for every pending file path to be calculated, and on systems with a 50ms response time to this call (I presume because of NAS/RAID-style gubbins), it meant opening the file export window could take minutes (issue #1413)
  • APNG metadata parsing no longer requires FFMPEG, greatly accelerating their import
  • I think I fixed the root cause of a weird bug we encountered and hacked around a couple weeks ago, where if a certain sort of downloaded page produced nothing via its parser, and it was detected initially as actually a valid file to import, but then that file import failed (e.g. ffmpeg went full bananas and thought a json file was an mp4), the import attempt would loop. the error handling now catches the unusual import failure gracefully, and the import object should be set to 'skipped' appropriately
  • fixed a harmless but annoying desync error popup that sometimes occurs when deleting a repository service

  • misc boring notes

  • to deal with the deferred delete system clashing with SQLite not allowing index renames, I moved the database index testing and creation system to a dynamic name format. it works but is a little hacky, so maybe we'll move to direct sqlite_master interrogation in future
  • unfortunately, the table shrink method I had planned to employ was not feasible (I wanted to do 'delete n rows', but it turns out that isn't compiled by default in all normal SQLite releases wew). I then experimented with several other strategies and settled on the KISS of 'select n, delete these n' in two queries, which worked out far better than my cleverer attempts anyway. the thing doesn't use much CPU time, and it cautiously autothrottles itself, and I've tested it in a bunch of situations, and I'm super happy with the performance, but if you do happen to get noticeable bumps of lag, most likely in PTR removal when the current_mappings giga-table is shrunk, turn off all database maintenance under the menu, for both idle and normal time, and let me know, and we'll figure it out
  • refactored APNG parsing code to the new 'HydrusAnimationHandling.py' and took out the ffmpeg code. now OpenCV/PIL figures out the resolution

• version 537

  • new filetype selector

  • I rewrote the expanding checkbox list that selects filetypes in 'system:filetype' and File Import Options into a more normal tree view with checkboxes. it is more compact and scrolls neatly, letting us stack it with all these new filetypes we've been adding and more in future. the 'clicking a category selects all children' logic is preserved
  • I re-ordered the actual filetypes in each sublist here. I tried to put the most common filetypes at the top and listed the rest in alphabetical order below, going for the best of both worlds. you don't want to scroll down to find webm, but you don't want to hunt through a giant hydev-written 'popularity' list to find realmedia either. let's see how it works out
  • I split all the archive types away from 'applications' into a new 'archives' group
  • and I did the same for the 'image project files' like krita and xcf. svg and psd may be significantly more renderable soon, so this category may get further shake-up
  • this leaves 'applications' as just flash and pdf for now
  • it isn't a big deal, but these new groups are reflected in _options->media_ too
  • all file import options and filetype system predicates that previously said 'all applications' _should_ now say 'all applications, image project files, or archives'

  • fast database delete

  • I have long planned a fix for 'the PTR takes ages to delete' problem. today marks the first step in this
  • deleting a huge service like the PTR and deleting/resetting/regeneratting a variety of other large data stores are now essentially instant. the old tables are not deleted instantly, but renamed and moved to a deferred delete zone
  • the maintenance task that actually does the deferred background delete is not yet ready, so for now these jobs sit in the landing zone taking up their original hard disk space. I expect to have it done for next week, so bear with me if you need to delete a lot this week
  • as this system gets fleshed out, the new UI under _database>db maintenance->review deferred delete table data_ will finish up too

  • misc

  • fixed a bitrot issue in the v534 update code related to the file maintenance manager not existing at the time of db update. if you got the 'some exif scanning failed to schedule!' popup on update, don't worry about it. everything actually worked ok, it was just a final unimportant reporting step that failed (issue #1414)
  • fixed the grid layout on 'migrate tags', which at some point in the recent past went completely bananas
  • tightened up some of the code that calculates and schedules deferred physical file delete. it now catches a couple of cases it wasn't and skips some work it should've
  • reduced some overhead in the hover window show/hide logic. in very-heavy-session clients, this was causing significant (7ms multiple times a second) lag
  • when you ok the 'manage login scripts' dialog, it no longer re-links new entries for all those scripts into the 'manage logins' system. this now only happens once on database initialisation
  • the manage login scripts test routine no longer spams test errors to popup dialogs. they are still written to log if you need more data
  • silenced a bit of PIL warning logspam when a file with unusual or broken EXIF data is loaded
  • silenced the long time logspam that oftens happens when generating flash thumbnails
  • fixed a stupid typo error in the routine that schedules downloading files from file repositories
  • `nose`, `six`, and `zope` are no longer in any of the requirements.txts. I think these were needed a million years ago as PyInstaller hacks, but the situation is much better these days

• version 536

  • more new filetypes

  • thanks to a user, we have XCF and gzip filetype support!
  • I rejiggered the new SVG support so there is a firmer server/client split. the new tech needs Qt, which broke the headless Docker server last week at the last minute--now the server has some sensible stubs that safely revert to the default svg thumb and give unknown resolution, and the client patches in full support dynamically
  • the new SVG code now supports the 'scale to fill' thumbnail option

  • misc

  • I fixed the issue that was causing tags to stay in the tag autocomplete lookup despite going to 0 count. it should not happen for new cases, and **on update, a database routine will run to remove all your existing orphans. if you have ever synced with the PTR, it will take several minutes to run!**
  • sending the command to set a file as the best in its duplicate group now presents a yes/no dialog to confirm
  • hitting the shortcut for 'set the focused file as better than the other(s)' when you only have one file now asks if you just want to set that file as the best of its group
  • fixed an erroneous 'cannot show the best quality file of this file's group here' label in the file relationships menu--a count was off
  • fixed the 'set up a hydrus.desktop file' setup script to point to the new hydrus_client.sh startup script name
  • thanks to a user, a situation where certain unhandled URLs that deliver JSON were parsing as mpegs by ffmpeg and causing a weird loop is now caught and stopped. more investigation is needed to fix it properly

  • boring stuff

  • when a problem or file maintenance job causes a new file maintenance job to be queued (e.g. if the client in a metadata scan discovers the resolution of a file was not as expected, let's say it now recognises EXIF rotation, and starts a secondary thumbnail regen job), it now wakes the file maintenance manager immediately, which should help clear out and update for these jobs quickly when you are looking at the problem thumbnails
  • if you have an image type set to show as an 'open externally' button in the media viewer, then it is now no longer prefetched in the rendering system!
  • I added a very simple .editorconfig file for the project. since we have a variety of weird files in the directory tree, I've made it cautious and python-specific to start with. we'll expand as needed
  • I moved the similar files search tree and maintenance tracker from client.caches.db to client.db. while the former table is regeneratable, it isn't a cache or precomputation store, _per se_, so I finally agreed to move it to the main db. if you have a giganto database, it may take an extra minute to update
  • added a 'requirements_server.txt' to the advanced requirements.txts directory, just for future reference, and trimmed the Server Dockerfile down to reflect it

  • client api

  • thanks to a user, fixed a really stupid typo in the Client API when sending the 'file_id' parameter to set the file
  • wrote unit tests for file_id and file_ids parameters to stop this sort mistake in future
  • if you attempt to delete a file over the Client API when one of the given files is delete-locked (this is an advanced option that stops deletion of any archived file), the request now returns a 409 Conflict response, saying which hashes were bad, and does not delete anything
  • wrote a unit test to catch the new delete lock test
  • deleted the old-and-deprecated-in-one-week 'pair_rows' parameter-handling code in the set_file_relationships command
  • the client api version is now 49

• version 535

  • misc

  • thanks to a user, we now have Krita (.kra, .krz) support! it even pulls thumbnails!
  • thanks to another user, we now have SVG (.svg) support! it even generates thumbnails!
  • I think I fixed a comparison statement calculator divide-by-zero error in the duplicate filter when you compare a file with a resolution with a file without one

  • petitions overview

  • _this is a workflow/usability update only for server janitors_
  • tl;dr: the petitions page now fetches many petitions at once. update your servers and clients for it all to work right
  • so, the petitions page now fetches lots of petitions with each 'fetch' button click. you can set how many it will fetch with a new number control
  • the petitions are shown in a new multi-column list that shows action, account id, reason, and total weight. the actual data for the petitions will load in quickly, reflected in the list. as soon as the first is loaded, it is highlighted, but double-click any to highlight it in the old petition UI as normal
  • when you process petitions, the client moves instantly to the next, all fitting into the existing workflow, without having to wait for the server to fetch a new one after you commit
  • you can also mass approve/deny from here! if one account is doing great or terrible stuff, you can now blang it all in one go

  • petitions details

  • the 'fetch x petition' buttons now show `(*)` in their label if they are the active petition type being worked on
  • petition pages now remember: the last petition type they were looking at; the number of petitions to fetch; and the number of files to show
  • the petition page will pause any ongoing petition fetches if you close it, and resume if you unclose it
  • a system where multi-mapping petitions would be broken up and delivered in tags with weight-similar chunks (e.g. if would say 'aaa for 11 files' and 'bbb in 15 files' in the same fetch, but not 'ccc in 542,154 files') is abandoned. this was not well explained and was causing confusion and code complexity. these petitions now appear clientside in full
  • another system, where multi-mapping petitions would be delivered in same-namespace chunks, is also abandoned, for similar reasons. it was causing more confusion, especially when compared to the newer petition counting tech I've added. perhaps it will come back in as a clientside filter option
  • the list of petitions you are given _should_ also be neatly grouped by account id, so rather than randomly sampling from all petitions, you'll get batches by user x, y, or z, and in most cases you'll be looking at everything by user x, and y, and then z up to the limit of num petitions you chose to fetch
  • drawback: since petitions' content can overlap in complicated ways, and janitors can work on the same list at the same time, in edge cases the list you see can be slightly out of sync with what the server actually has. this isn't a big deal, and the worst case is wasted work as you approve the same thing twice. I tried to implement 'refresh list if count drops more than expected' tech, but the situation is complicated and it was spamming too much. I will let you refresh the list with a button click yourself for now, as you like, and please let me know where it works and fails
  • drawback: I added some new objects, so you have to update both server and client for this to work. older/newer combinations will give you some harmless errors
  • also, if your list starts running low, but there are plenty more petitions to work on, it will auto-refresh. again, it won't interrupt your current work, but it will fetch more. let me know how it works out
  • drawback: while the new petition summary list is intentionally lightweight, I do spend some extra CPU figuring it out. with a high 'num petitions to fetch', it may take several seconds for a very busy server like the PTR just to fetch the initial list, so please play around with different fetch sizes and let me know what works well and what is way too slow
  • there are still some things I want to do to this page, which I want to slip in the near future. I want to hide/show the sort and 'num files to show' widgets as appropriate, figure out a right-click menu for the new list to retry failures, and get some shortcut support going

  • boring code cleanup

  • wrote a new petition header object to hold content type, petition status, account id, and reason for petitions
  • serverside petition fetching is now split into 'get petition headers' and 'get petition data'. the 'headers' section supports filtering by account id and in future reason
  • the clientside petition management UI code pretty much got a full pass
  • cleaned a bunch of ancient server db code
  • cleaned a bunch of the clientside petition code. it was a real tangle
  • improved the resilience of the hydrus server when it is given unacceptable tags in a content update
  • all fetches of multiple rows of data from multi-column lists now happen sorted. this is just a little thing, but it'll probably dejank a few operations where you edit several things at once or get some errors and are trying to figure out which of five things caused it
  • the hydrus official mimetype for psd files is now 'image/vnd.adobe.photoshop' (instead of 'application/x-photoshop')
  • with krita file (which are actually just zip files) support, we now have the very barebones of archive tech started. I'll expand it a bit more and we should be able to improve support for other archive-like formats in the future

• version 534

  • user submissions

  • thanks to a user, we now have SAI2 (.sai2) file support!
  • thanks to a user, the duplicate filter now says if one file has audio. this complements the recent Hydrus Video Deduplicator (https://github.com/appleappleapplenanner/hydrus-video-deduplicator), which can queue videos up in your dupe filter
  • thanks to a user, we now have some nice svg images in the help->links(?) menu instead of gritty bitmaps
  • thanks to a user, some help documentation for recent client vs hydrus_client changes got fixed

  • quality of life/new stuff

  • the media viewer's top-area 'removed from x' lines for files deleted from a local file service no longer appear--unless that file is currently in the trash. on clients with busy multiple local file services, they were mostly just annoying and spammy. if you need this data, hit up the right-click menu of the file--it is still listed there
  • the 'loading' media page now draws a background in the same colour as the thumbnail grid, so new searches or refreshes will no longer flash to a default grey colour--it should just be a smooth thumbs gone/thumbs back now
  • added a new shortcut action, 'copy small bmp of image for quick source lookups', for last week's new bitmap copy action
  • it turns out PNG and WEBP files can have EXIF data, and our existing scanner works with them, so the EXIF scanner now looks at PNGs and WEBPs too. PNGs appear to be rare, about 1-in-200. I will retroactively scan your existing WEBPs, since they have EXIF more commonly, maybe as high as 1-in-5, and are less common as a filetype anyway so the scan will be less work, but when you update you will get a yes/no dialog asking if you want to do PNGs too. it isn't a big deal and you can always queue it up later if you want

  • fixes

  • I banged my head against the notes layout code and actually had great success--a variety of borked note-spilling-over-into-nothing and note-stretching-itself-crazy and note-has-fifty-pixels-of-margin issues are fixed. let me know if you still have crazy notes anywhere
  • the duplicate filter right-hand hover is now more aggressive about getting out of the way of the notes hover, especially when the notes hover jitter-resizes itself a few extra pixels of height. the notes hover should no longer ever overlap the duplicate filter hover's top buttons--very sorry this took so long
  • when you drag and drop thumbnails out of the program while using an automatic pattern to rename them (_options->gui_), all the filenames are now unique, adding '(x)' after their name as needed for dedupe. previously, on duplicates, it was basically doing a weird spam-merge
  • fixed an issue when sanitizing export filenames when the destination directory's file system type cannot be determined
  • fixed a bug when doing a search in a deleted file domain with 'file import time' as the file sort
  • fixed a bug when hitting the shortcut for 'open file in media viewer' on a non-local file
  • fixed a bug when the client wants to figure out view options for a file that has mime 'application/unknown'
  • I may have improved the 'woah the db caches are unsynced' error handling in the 'what siblings and parents does this have?' check when you right-click tags

  • weird bitmap pastes

  • fixed the new 'paste image' button under `system:similar files` for a certain category of unusual clipboard bitmaps, including several that hydrus itself generates, where it turns out the QImage storage system stores extra padding bytes on each line of pixels
  • fixed the new 'paste image' button when the incoming bitmap has a useless alpha channel (e.g. 100% transparent). this was not being stripped like it is for imported images, and so some similar files data was not lining up
  • many bitmaps copied from other programs like Firefox remain slightly different to what hydrus generates (even though both are at 100% scale). my best guess here is that there is some differing ICC-profile-like colour adjustment happening somewhere, probably either a global browser setting, the browser obeying a global GPU setting, a simply better application of such image metadata on the browser's side, or maybe a stripping of such data, since it seems a 'copy image' event in Firefox also generates and attaches to your clipboard a temporary png file in your temp folder, so maybe the bitmap that we pull from the clipboard is actually generated during some conversion process amidst all that, and it loses some jpeg colour data. whatever the case here, it changes the pixel hash and subtly alters the perceptual hash in many cases. I'm bumping the default distance on this search predicate up to 8 now, to catch the weirder instances

  • misc

  • the 'does the db partition have 500MB free?' check that runs on database boot now occurs after some initial database cleanup, and it will use half the total database size instead, if that is smaller than 500MB, down to 64MB (issue #1373)
  • added a note to the 'running from source' help that the newer mpv dll seems to work on Qt5 and Windows 7 (issue #1338)
  • the twitter parsers and gugs are removed from the defaults for new users. a shame, but we'll see what happens in future
  • more misc linting cleanup

  • ratings on the client api

  • the services object now shows `star_shape` and `min_stars` and `max_stars` for like/dislike and numerical rating services
  • the file metadata object now has a 'ratings' key, which lists `rating_service_key->rating` for all the client's rating services. this thing is simple and uses human-friendly values, but it can hold several different data types, so check the help for details and examples
  • a new permission, 'edit ratings', is added.
  • a new command, `/edit_ratings/set_rating`, is added. Guess what it does! (issue #343)
  • the help is updated for these
  • the unit tests are updated for these
  • the client api version is now 48

• version 533

  • macOS App crashes

  • unfortunately, last week's eventFilter work did not fix the macOS build's crashing--however, thanks to user help, we figured out that it was some half-hidden auxiliary Qt library that updated in the background starting v530 (the excellently named `PyQt6-Qt6` package). the build script is updated to roll back this version and it seems like things are fixed. this particular issue shouldn't happen again. sorry for the trouble, and let me know if there are any new issues! (issue #1379)

  • misc

  • the download panels in subscription popup windows are now significantly more responsive. ever since the popup manager was embedded into the gui, popup messages were not doing the 'should I update myself?' test correctly, and their network UI was not being updated without other events like surrounding widgets resizing. I was wondering what was going on here for ages--turns out it was regular stupidity
  • if an image has width or height > 1024, the 'share->copy' menu now shows a second, 'source lookup' bitmap, with the resolution clipped to 1024x1024
  • 'sort files by hash' can now be sorted asc or desc. this also fixes a bug where it was secretly either sorting asc or desc based on the previous selection. well done to the user who noticed and tested this
  • if system:limit=0 is in a search, the search is no longer run--it comes back immediately empty. also, the system:limit edit panel now has a minimum value of 1 to dissuade this state
  • the experimental QtMediaPlayer now initialises with the correct volume/mute and updates on volume/mute events. the scanbar and volume control UI are still hidden behind the OpenGL frame for now, but one step forward
  • the system that caches media results now hangs on to the most recent 2048 files aggressively for two minutes after initial load. previously, if you refreshed a page of unique files, or did some repeated client api work on files that were not loaded somewhere as thumbs, in the interim periods those media objects were not strictly in non-weak memory anywhere in the client and could have been eligible for clearing out of the cache. now they are a bit more sticky
  • added some info on editing predicates and the various undocumented click shortcuts the taglist supports (e.g. ctrl+double-left-click) to the 'getting started with searching and sorting' help page
  • added a link to the Client API help for 'Hydrus Video Deduplicator' (https://github.com/appleappleapplenanner/hydrus-video-deduplicator), which neatly discovers duplicate videos in your client and queues them up in the duplicate filter by marking them as 'potential dupes'

  • sub-gallery url network parsing gubbins

  • sub-gallery import objects now get the tags and custom headers that are parsed with them. if the sub-gallery urls are parsed in 'posts' using a subsidiary parser, they only inherit the metadata within their post
  • sub-gallery import objects now use their parent gallery urls as referral header
  • sub-gallery import objects now inherit the 'can generate more pages' state of their parents (previously it was always 'yes')
  • 'next page' gallery urls do not get the tags they are parsed with. this behaviour makes a little less sense, and I suspect it _could_ cause various troubles, so I'll wait for more input, bug reports, and a larger cleanup and overhaul of how metadata is managed and passed down from one item to the next in the downloader system
  • generally speaking, when file and gallery import objects have the opportunity to acquire tags or headers, they'll now add to the existing store rather than replace it. this should mean if they both inherit and parse stuff, it won't all overwrite each other. this is all a giant mess so I have a cleanup overhaul planned

  • boring stuff

  • if a critical drive error occurs (e.g. running out of space on a storage drive), the popup message saying 'hey everything just got mega-paused' is now a little clearer about what has happened and how to fix it
  • similarly, the specific 'all watchers are paused'-style messages now specifically state 'network->pause to resume!' to point users to this menu to fix this tricky issue. this has frustrated a couple of newer users before
  • to reduce confusion, the 'clear orphan files' pre-job now only presents the user one combined dialog
  • improved how pages test and recognise that they have changes and thus should be saved--it works faster, and a bunch of false negatives should be removed
  • improved the safety routine that ensures multiple-column list data is read-only
  • fixed .txt sidecar importers' description labels, which were missing extra text munging
  • to relieve some latency stress on session load, pages that are loading their initial files in the background now do so in batches of 64 rather than 256
  • fixed some bad error handling in the master client db update routine
  • fixed a scatter of linting problems a user found
  • last week's pixiv parser hotfix is reinforced this week for anyone who got the early 532 release
  • made some primitive interfaces for the main controller and schedulable job classes and ensured the main hydrusglobals has type hinting for these--now everything that talks to the controller has a _bit_ of an idea what it is actually doing, and as I continue to work on this, we'll get better linting
  • moved the client DataCache object and its friends to a new 'caches' module and cleaned some of the code

• version 532

  • misc

  • whenever you say 'show these files in a new page', the new page now has a search interface. it starts with a 'system:hash' pre-populated with the files' hashes, so you can now easily narrow down or return to the stuff you are playing with! original file sort order is preserved until you alter or refresh the search
  • tags' `right-click->search` menu now has a 'open in a new duplicate filter' for quick spawning of duplicate filters for specific searches
  • the duplicate filter no longer flicks to the 'preparation' tab if there is work to do on the first numbers fetch. this thing has been driving me nuts, I don't know why I wrote it that way to begin with
  • improved the reliability of certain session object saving--I believe some situations where the 'searching immediately' and 'this search was completed' status where not being saved for some page queries. this _may_ solve a long time bug where some pages would refresh on load
  • all search pages that load with files now explicitly reaffirm internally that they are starting with a completed search, which should reduce some related edge case buggy behaviour here
  • the 'string to string' edit control now tries to compensate if it is incorrectly given non-string data. somewhere in the html parsing formula UI this happened, an integer sneaking in the key/value of the tag rule, maybe by manual human JSON editing, but I'm not really sure. should be handled correctly now though. let me know if you are into this and discover anything
  • every 'eventFilter' in the program now catches Exceptions ruthlessly. it turns out Qt can't handle an Exception escaping one of these, and this _may_ be the cause of some >=v530 crashing on macOS related to multi-column list interaction under issue #1379. it is probably the cause of some other crashes that I haven't been able to figure out--these will now give normal popup errors, so let me know if you see anything. if you have had crazy crashes in macOS recently and these changes don't fix you, reverting back to v529 is apparently ok! there have been no big database updates in that time, so you should be able to just install v529 on your existing install and be off
  • the routine that purges files from the trash now uses fewer database queries to find eligable files. some Linux guys have been working with me on memory explosions possibly in this area--let me know if you notice any difference
  • the 'clear trash' command in review services is politer to your database, breaking up a large amount of trash into smaller groups
  • the program no longer moans to the log when it physically deletes a file and files no accompanying thumbnail to delete--this is true for several situations, and not worth the logspam
  • fixed a typo error in the `url class links` 'try to fill in the gaps' command

  • pixiv downloader

  • I reworked the pixiv parser changes from a couple weeks ago. as background, what happened is pixiv said if you aren't logged in, you can't get the 'original' quality of the file any more. my first fix was to say 'ok, if the user is not logged in, get the lower quality', but this was the wrong decision. the parser now vetoes, causing an 'ignored' result and telling you the problem in the import note. if you _do_ want to get the lower quality image and not log in, this is now selectable as an alternate parser under _network->downloader components->manage url class links_
  • also, a variety of old pixiv objects and other experiments are deleted and merged today. the parsers that worked on the old html format, `pixiv manga page parser`, `pixiv manga_big page parser`, `pixiv single file page parser - new layout`, and `pixiv tag search gallery page parser` will be deleted from your client, and the old gallery url class, `pixiv tag search gallery page` meets a similar fate. `pixiv manga_big page` and `pixiv manga page` are removed and their urls merged into a more accomodating `pixiv file page`, which stays to hold all the legacy pixiv URLs, which on the site are automatically redirected to the new format. thanks to a user for helping me with what here was cruft (issue #947)

  • mpv logging and emergency halt

  • a user sent me a cool truncated twitter video download that, when loaded into mpv, would crash the program after a click or two around the player. this sent me on an odyssey into the mpv logging system and event loop and some really bizarre behaviour under the hood, and, long story short, mpv will notice this particular problem class in future and immediately unload the file and present the user with a dialog explaining the issue. it also won't let you load that file again that boot
  • to recognise this error class, I broaden what is logged and scan the lines as they come in. I've been careful in how I filter, but it may produce some false positives. let me know if this thing triggers for any files that seem fine in an external player
  • errors of unknown severity are now printed silently to the log with a little intro text saying which file it was and so on. there are a bunch of these with the sorts of files we deal with, stuff like missing chapter marks or borked header data. I expect I'll work on silencing the ones we confirm are no big deal, but if you encounter a ton of them, particularly if you know some cause crashes, please now check your log and let me know what you see
  • if you have two mpv players playing media at the same time, this reporting system will report the info for both files--sorry, I had to hack this gubbins! future versions of mpv or python-mpv may open some doors here

  • client api

  • the `/get_files/file` command now has a `download=true` parameter which converts the `Content-Disposition` from `inline` (show the file) to `attachment` (auto-download or open save-as dialog) (issue #1375)
  • added help and a unit test for the above
  • client api version is now 47

• version 531

  • misc

  • fixed editing favourite searches, which I accidentally broke last week with the collect-by updates
  • when you right-click a tag and get the siblings/parents menus, the list of copyable siblings, parents, and children is now truncated to 10 items each per service. stuff like pokemon has hundreds of children and for a very long time has been spamming giganto 11-column menus that cover the entire screen
  • same menu truncation for the open/copy URLs menu. if there's a file that has 600 URLs for interesting technical reasons, it won't nuke you any more (issue #1037)
  • updated the default pixiv file page parser, which recently broke for users who were not logged in. they seem to hide original size behind the login now, so if you do a lot of pixiv work, get Hydrus Companion or figure out a cookies.txt solution and get yourself logged in
  • the downloader progress panels have a couple of status text improvements: first, they will stop saying 'waiting for a work slot' when the actual error is something unusual such as the gallery search hitting the file limit. second, when there is an unusual status and the downloader is in the paused state, it can now properly differentiate between 'paused' and 'pausing'
  • some invalid URL strings now raise the correct error in the downloader system, causing them to be properly filtered away instead of sticking around and being unhelpful
  • if there is a connection error because of an SSL issue, the network job is now retried like any other connection error. I originally thought these were all non-retryable like cert validation errors, but it seems some of them are just write timeouts etc.. during the negotiation, so let's see how it goes
  • I believe I have fixed an error when selecting a tag in a list when that list had been previously shift-selected and then cleared and repopulated
  • manage siblings and parents should be better about focusing the correct text input after they boot and load
  • in future, if a taglist tries to deselect something it no longer has, it'll do an emergency 'deselect all' to exorcise the ghosts fully
  • reworded the text around 'reset potential duplicates' action in the duplicates page to be more clear on what it does
  • I tinkered with some of the shutdown code hoping to catch an odd issue of the exit 'last session' not saving correctly, but I don't think I figured the issue out. if you have noticed you boot up and get a session that missed up to the last 15 minutes of changes before you last shut down, please let me know you your details
  • added a link to `tagrank`, a new Client API project at https://github.com/matjojo/tagrank, to the Client API help. it shows you pairs of comparison images over and over and uses `trueskill` ranking algorithm to figure out which tags are your favourite
  • added a link to 'Send to Hydrus', a Client API project at https://github.com/Wyrrrd/send-to-hydrus, to the Client API help. it sends URLs from an Android device to your client

  • client api

  • as part of a plan to migrate to service_key indexing everywhere and reduce file_metadata bloat, the client api has a new `services` structure, a service information Object where `service_key` is the key. this is now in the `/get_services` call and `/get_files/file_metadata`, under `services` under the root. the old type-based structure in `/get_services` and the in-file embedding of service info in `/get_files/file_metadata` are still in place, so nothing breaks today, but I am officially declaring them deprecated, to be deleted in 2024, and recommend all Client API devs move to the new system before the new year
  • the new service object also includes info on the local rating services. I'd like to add ratings to file_metadata fairly soon
  • if you don't want the services object in `/get_files/file_metadata`, there's a new `include_services_object` param you can set to false to hide it
  • updated the unit tests and client api help to reflect all this. main new section: https://hydrusnetwork.github.io/hydrus/developer_api.html#services_object
  • the client api version is now 46

  • update woes

  • I somewhat successfully pounded my head against an issue where the first tab (usually 'my tags') was disappearing in the _manage tags/siblings/parents_ dialogs for some users. this bug, for real, seems to be the combination of (Python 3.11 + PyQt6 6.5.x + two tabs + total tab text characters > ~12 + tab selection is set to 1 during init event). Change any of those things and it doesn't happen. This is so weird a problem to otherwise normal code that I won't pivot all my 50-odd instances of tab selection to handle it and instead have hacked an answer for the three tag dialogs and filename tagging. Sorry for the trouble if you got this! Let me know if you see any more
  • in a similar-but-different thing, PySide6 6.5.1 has a bug related to certain Signal connections. don't use it with hydrus, it messes up all my menus! their dev notes suggest they are going to have a fix/revert for 6.5.1.1

• version 530

  • autocomplete and system predicates

  • the normal autocomplete text input in file search pages now parses system tags if you type them! For a long time, this cool system has only been awkwardly available, but now it should work straight out of the box. not every predicate is supported, and sometimes what parses is slightly different to what you see, but I am improving things regularly, so let me know what doesn't work
  • the normal autocomplete text input in file search pages now has a paste button! it takes tags in the normal newline-separated hydrus format and is plugged into the system predicate parser too. it should obey the same rules as if you were typing, so if you put in a negated tag, or a wildcard or namespace wildcard, and that's allowed with your current settings, it'll propagate. anything that isn't allowed or won't parse correctly is skipped silently for now
  • the system predicate parser now supports the new 'similar to data' similar files search added last week. there isn't an easy way to generate the pixel and perceptual hashes yet (this will come soon to the Client API), but if you have the hashes, the thing should now parse. same format as the existing 'similar to( files)', but just say 'similar to data' and mix and match the 64- and 16-character hashes and it'll figure it out
  • fixed system predicate parsing for 'system:has note with name xxx', which was parsing as a borked 'system:has note(s)', and the same deal for 'has no note'
  • also made the 'system:has/no notes' and 'system:has a note named xxx' more flexible. they can take more english variants of the phrase, and if you give a note name in "quotes" (e.g. if you copy the system predicate string and paste it back in), it'll strip them

  • misc

  • highlighting a gallery downloader or thread watcher is now asynchronous! this means if you load up a meaty uncached 3,000-strong downloader, the client will no longer lock up for a few seconds--it'll load the files in the background, in 256-file chunks like a normal search page, and then present them when ready. while in the loading state, the to-be-highlighted downloader will be prepended with `> ` instead of `* `, and its loading is completely cancellable--you can unhighlight it or highlight something else and the ongoing job will promptly cancel and let the new one start. if a loading job takes more than three seconds, it will make a popup window with its ongoing progress, which also has a cancel button
  • when you say to 'open files in a new page', the current file sort and collect is copied to the new page, and if you have a collect set, the new page will collect
  • when parsing URLs and attempting to match relative URLs (''/post/123456') to the original domain ('example.com'), if that join fails, it now just adds the parsed text. this should stop borked errors from halting the whole parse (e.g. mysterious 'Invalid IPv6 URL' error, which was probably an errantly parsed open square bracket) while also helping debugging
  • improved URL-repairing in parsing. it trims gumpf before a recognisable URL (`title - https://example.com/123456`) is now more precise, and instances of weird scheme-spam (`https://http://example.com/123456`) are now fixed for mixes of schemes and replaced with the final scheme
  • the thumbnail duplicate files menu now tries to recognise if the king of a group has been deleted and will say so rather than 'show the best quality file of this file\'s group'
  • if you open some duplicate files from the right-click menu (e.g. show 'king') and the search can't find them, it now searches "all known files" as a backup and tells you in a popup if the backup worked or if it just couldn't find anything

  • some boring cleanup

  • refactored the media controller (which drives every page in the client) and the media controller panel (the actual UI) code into separate files; now the various other guys that look at the controller have proper typing and inheritance, and all the thumbnail grids are now explicitly told their respective media controllers and have better access to stuff like the current sort
  • the sort widget no longer hangs onto the media controller--it just communicates changes through Qt signals
  • same doubly so for the collect widget, which no longer has a mickey-mouse pubsub chain and just Qt signals its stuff now
  • misc page code and sort/collect code cleanup, multiple orphaned pubsubs removed
  • moved ClientSearch and ClientSearchParseSystemPredicates to a new 'search' module
  • spun off the autocomplete parsing and result caching code into a new ClientSearchAutocomplete
  • added a heap of note system predicates to the system pred parsing unit tests, and some for the new 'similar to data' too
  • updated the `requests` in the requirements.txts up from 2.28.1 to 2.31.0 due to some security vulnerability related to `Proxy-Authorization` headers and in-url user/pass authentication when redirecting to an https destination. I don't think we used that stuff (unless the proxy settings cause it to happen under the hood), but let's update anyway. if you run from source, you might like to run setup_venv again

• version 529

  • similar files search

  • hydrus now supports a 'SauceNAO'-style workflow on its own files, quickly looking up if you have something that looks like the given file, without having to import it, using a new variant of the 'system:similar to' search predicate. just open up the new 'system:similar files' entry, which now has two tabs, and on the first just paste image data or a file path from your clipboard and it'll calculate the data for you
  • similar files also gets a search cache this week. this makes all repeat searches massively faster, helps out successive searches (e.g. the same file at 0, 4, then 8 distance), and should accelerate all maintenance search by a good bit depending on the size and shape of your database (on my test database of only ~10k files, it sped things up 3-4x)
  • 'system:similar to' search predicates are no longer mutually exclusive in the same search--you can now have multiple
  • cleaned up a bunch of the similar files code generally. the main search function is split into pieces and common calls are spun off into their own thing

  • misc

  • added a new shortcut action, 'open file in file explorer', which opens the file in your file Explorer. if you haven't used this before, it only works on Windows and macOS and can be buggy. on Windows, if the explorer takes too long to open, it won't select the file correctly, so hit it again
  • thanks to a user, the html parsing formula can now search in a sideways direction, either finding the previous or following sibling html tags (as opposed to just search descendants/ancestors)
  • if an export folder is set to 'synchronise' and also needs to delete some symlinks (either it regularly makes symlinks, or it is clearing symlinks from an old run), _and_ those symlinks now point to since-deleted files, the dead symlinks should now delete correctly! thanks for an interesting report here
  • the docker build now has pympler support for memory profiling. note that this does not work very well--it is unfathomably laggy atm for any client of real size, so bear with me
  • the new Qt Media Player experiment is now more careful about how it deletes old windows. old players are handed off to the main gui, which takes ownership and explicitly waits for them to finish current work, then asks them to unload their media, and then, only when they are all clear sends the window delete signal. this should stop some READY/NULL errors people were seeing on unload, and hopefully without causing new stability problems (I've had crash trouble with explicitly unloading media before destroy before, but I'm doing it super safe here, so we'll see)
  • I added some more error reporting to the related area in the mpv player--if it fails to unload a media, it now prints the details to log--let's see if we can improve this too
  • when files fail to import for reasons other than veto or unsupported file, they now say the actual exception type in their first line summary

  • client api

  • when the api sends a file to be imported and it fails, the response 'note' now just has this human-readable top level line (it used to have the full error trace), and a new entry 'traceback' has the trace
  • the client api version is now 45

  • future build

  • to improve library update testing, I have set up a second, 'future' build that is the same as a normal release but uses newer library versions, for instance Python 3.10 from 3.9 and Qt 6.5.0 rather than 6.4.1. I am not sure how often I will be making this build--I don't want to spam, so I'm thinking once per month, but maybe we'll ultimately end up incorporating it into the main build and just kick it out every week--but please feel free to test them out as they do happen and let me know if you encounter any problems booting or with anything else. the idea here is to get more user situations, particularly older OSes, testing pending library updates so I can be more confident about pulling the trigger on moving up in the master build (the recent jump to Qt 6.4.1 caused several Win 10 users to have an annoying 2-second delay on opening any new search page, but 6.5.0 doesn't have this, so if you encountered this error, please try this build and let me know how it goes). the build is in the normal github releases stream, marked as a pre-release. v528-future is here: https://github.com/hydrusnetwork/hydrus/releases/tag/v528-future-1

• version 528

  • faster file search cancelling

  • if you start a large file search and then update or otherwise cancel it, the existing ongoing search should stop a little faster now
  • all timestamp-based searches now cancel very quickly. if you do a bare 'all files imported in the last six months' search and then amend it with 'system:inbox', it should now update super fast
  • all note-based searches now cancel quickly, either num_notes or note_names
  • all rating-based searches now cancel quickly
  • all OR searches cancel faster
  • and, in all cases, the cancel tech works a little faster by skipping any remaining search more efficiently
  • relatedly, I upgraded how I do the query cancel tech here to be a bit easier to integrate, and I switched the 20-odd existing cancels over to it. I'd like to add more in future, so let me know what cancels slow!

  • system predicate parsing

  • the parser is more forgiving of colons after the basename, e.g. 'system:import time: since 2023-01-01' now parses ok
  • added 'since', 'before', 'around', and 'day'/month' variants to system datetime predicate parsing as more human analogues of the '>' etc... operators
  • you can now say 'imported', 'modified', 'last viewed', and 'archived' without the 'time' part ('system:modified before 2020-01-01')
  • also, 'system:archived' with a 'd' will now parse as 'system:archive'
  • you now can stick 'ago' ('system:imported 7 days ago') on the end of a timedelta time system pred and it should parse ok! this should fix the text that is copied to clipboard from timedelta system preds
  • the system predicate parser now handles 'file service' system preds when your given name doesn't match due to upper/lowercase, and more broadly when the service has upper case characters. some stages of parsing convert everything to lowercase, making this tricky, but in general it now does a sweep of what you entered and then a sweep that ignores case entirely. related pro-tip: do not give two services the same name but with different case

  • misc

  • you can now edit the default slideshow durations that show up in the media viewer right-click menu, under _options->media_. it is a bit hacky, but it works just like the custom zoom steps, with comma-separated floats
  • fixed 'system:num notes < x', which was not including noteless files (i.e. num_notes = 0) in the result
  • fixed a bug in _manage services_ when adding a local file service and then deleting it in the same dialog open. a test that checks if the thing is empty of files before the delete wasn't recognising it didn't exist yet
  • improved type checking when pasting timestamps in the datetime widget, I think it was breaking some (older?) versions of python

  • some more build stuff

  • fixed the macOS App, which was showing a 'no' symbol rather than launching due to one more thing that needed to be redirected from 'client' to 'hydrus_client' last week (issue #1367)
  • fixed a second problem with the macOS app (unlike PyInstaller, PyOxidizer needed the 'hydrus' source directory, so that change is reverted)
  • I believe I've also fixed the client launching for some versions of Python/PyQt6, which had trouble with the QMediaPlayer imports
  • cleaned up the PyInstall spec files a little more, removing some 'hidden-import' stuff from the pyinstaller spec files that was no longer used and pushing the server executables to the binaries section
  • added a short section to the Windows 'running from source' help regarding pinning a shortcut to a bat to Start--there's a neat way to do it, if Windows won't let you
  • updated a couple little more areas in the help for client->hydrus_client

• version 527

  • important updates

  • There are important technical updates this week that will require most users to update differently!
  • first, OpenCV is updated to a new version, and this causes a dll conflict on at least one platform, necessitating a clean install
  • second, the program executables are renamed from 'client' and 'server' to 'hydrus_client' and 'hydrus_server', necessitating shortcut updates
  • as always, but doubly so this week, I strongly recommend you make a backup before updating. the instructions are simple, but if there is a problem, you'll always be able to roll back
  • so, in summary, for each install type--
  • - if you use the windows installer, install as normal. your start menu 'hydrus client' shortcut should be overwritten with one to the new executable, so you don't have to do anything there, but if you use a custom shortcut, you will need to update that too
  • - if you use one of the normal extract builds, you will have to do a 'clean install', as here https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs . you also need to update your program shortcuts
  • - macOS users have no special instructions. update as normal
  • - source users, git pull as normal. if you haven't already, feel free to run setup_venv again to get the new OpenCV. update your launch scripts to point at the new 'hydrus_client.py' scripts
  • - if you have patched my code, particularly the boot code, obviously update your patches! the 'hydrus_client.py' scripts just under 'hydrus' module all got renamed to '_boot' too!
  • also, some related stuff like firewall rules (if you run the Client API) may need updating!

  • boring related update stuff

  • the Windows build's sqlite3.dll and exe command line interface are updated to the latest, 3.41.2
  • the 'updating' help now has a short section for the 526->527 update step, reiterating the above
  • the builds no longer include the hydrus source in the 'hydrus' subdirectory. this was an old failed test in dual-booting that was mostly forgotten about and now cleaned up. if you want to run from source, get the source
  • the windows hydrus_client and hydrus_server executables now have proper version info if you right-click->properties and look at the details tab

  • Qt Media Player

  • THIS IS VERY BUGGY AND SOMETIMES CRASHY; DISABLED FOR MOST USERS; NOT FOR NORMAL USE YET
  • I have integrated Qt's Media Player into hydrus. it is selectable in _options->media_ (if you are an advanced user and running from source) and it works like my native viewer or mpv. it has good pixels-on-screen performance and audio support, but it is buggy and my implementation is experimental. for some reason, it crashes instantly when running from a frozen executable, so it is only available for source users atm. I would like feedback from advanced source users who have had trouble with mpv--does it work? how well? any crashes?
  • this widget appears to be under active development by the Qt guys. the differences between 6.4.1 vs 6.5.0 are significant. I hope the improvements continue!
  • current limitations are:
  • - It is only available on Qt6, sorry legacy Qt5 source users
  • - this thing crashed the program like hell during development. I tightened it up and can't get it to crash any more with my test files on source, but be careful
  • - the video renderer is OpenGL and in Qt world that seems to mean it is ALWAYS ON TOP at all times. although it doesn't interfere with click events if you aim for the scanbar (so Qt's z-indexing logic is still correct), its pixels nonetheless cover the scanbar and my media viewer hover windows (I will have to figure out a different scanbar layout with this thing)
  • - longer audio-only files often stutter intolerably
  • - many videos can't scan beyond the start
  • - some videos turn into pixel wash mess
  • - some videos seem to be cropped wrong with green bars in the spare space
  • - it spams a couple lines of file parsing error/warning info to the log for many videos. sometimes it spams a lot continuously. no idea how to turn it off!
  • anyway, despite the bugs and crashing, I found this thing impressive and I hope it can be a better fallback than my rubbish native viewer in future. it is a shame it crashes when built, but I'll see what I can do. maybe it'll be ready for our purposes by Qt7

  • misc

  • if twisted fails to load, its exact error is saved, and if you try to launch a server, that error is printed to the log along with the notification popup

• version 526

  • there will be an important update next week

  • next week's release will have two important program changes--I will integrate an OpenCV update, which will require 'extract' users to perform a clean install, and the executables are finally changing from 'client' and 'server' to 'hydrus_client' and 'hydrus_server'! be prepared to update your shortcuts and launch scripts

  • time

  • fixed a stupid logical bug in my new date code, which was throwing errors on system:time predicates that had a month value equal to the current month (e.g. 'x years, 5 months' during May)--sorry! (issue #1362)
  • when a subscription dies, the popup note about it says the death velocity period in the neat '180 days', as you set in UI, rather than converting to a date and stating the number of months and days using the recent calendar calculation updates
  • I unified some more 'xxxified date' UI labels to be 'xxxified time'. we're generally moving to the latter format as the ideal while still accepting various combinations for system parsing input

  • shortcuts

  • added 'media play-pause/previous/next' and 'volume up/down/mute' key recognition to the shortcut system. if your keyboard/headphones have media keys, they _should_ be mappable now. note, however, that, at least on Windows, while these capture in the hydrus UI, they seem to have global OS-level hooks, and as far as I can tell Qt can't stop that event propagating, so these may have limited effectiveness if you also have an mp3 player open, since Windows will also send the 'next' call to that etc... it may be there is a nice way to properly register a Qt app as a media thing for Windows to global-hook these events to, but I'm not sure!
  • also added 'mouse task button' to the mappable buttons. this is apparently a common Mouse6 mapping, so if you have it, knock yourself out
  • the code in the shortcut system that tries to detect and merge many small scroll wheel events (such as the emulated scroll that a trackpad may generate) now applies to all mouse devices, not just synthesised events. with luck, this will mean that mice that generate like 15 smoothscroll events of one degree instead of one of fifteen degrees for every wheel tick will no longer spam-navigate the media viewer wew

  • misc

  • to save you typing/pasting time, the 'enter your reason' prompts in manage tags, tag siblings, and tag parents now remember the last five custom reasons you enter! you can change the number saved using the new option under _options->tags_, including setting it to 0 to disable the system
  • fixed pasting tags in the manage tags dialog when the number of tags you are pasting is larger than the number of allowed 'recent tags'. previously it was saying 'did not understand what was in the clipboard', so hooray for the new error reporting
  • every multi-column list in the program now has a 'reset column widths' item in its header right-click menu! when these reset events happen, the respective lists also resize themselves immediately, no restart required
  • when you set 'try again' on an import object, it now clears all saved hashes from the import object (including the SHA256 which may have been linked from the database in an 'already in db'/'previously deleted' result). this will ensure the next attempt is not poisoned by these hashes (which can happen for various reasons) in the subsequent attempt. basically 'try again' resets better now (issue #1353)

  • some build stuff

  • the main build script now only uses Node16 sub-Actions (Node12 support is deprecated and being dropped in June)
  • the main build script no longer uses set-output commands (these are deprecated and being dropped later in the year I think, in favour of some ENV stuff)
  • tidied some cruft from the main build script
  • I moved the 'new' python-mpv in the requirements.txts from 1.0.1 to 1.0.3. source users might like to rebuild their venvs again, particularly Windows users who updated to the new mpv dll recently

• version 525

  • library updates

  • after successful testing amongst source users, I am finally updating the official builds and the respective requirements.txts for Qt, from 6.3.1 to 6.4.1 (with 'test' now 6.5.0), opencv-python-headless from 4.5.3.56 to 4.5.5.64 (with a new 'test' of 4.7.0.72), and in the Windows build, the mpv dll from 2022-05-01 to 2023-02-12 (API 2.0 to 2.1). if you use my normal builds, you don't have to do anything special in the update, and with luck you'll get slightly faster images, video, and UI, and with fewer bugs. if you run from source, you might want to re-run your setup_venv script--it'll update you automatically--and if you are a modern Windows source user and haven't yet, grab the new dll here and rename it to mpv-2.dll https://sourceforge.net/projects/mpv-player-windows/files/libmpv/mpv-dev-x86_64-20230212-git-a40958c.7z . there is a chance that some older OSes will not be able to boot this new build, but I think these people were already migrated to being source users when Win 7-level was no longer supported. in any case, let me know how you get on, and if you are on an older OS, be prepared to rollback if this version doesn't boot
  • setup_venv.bat (Windows source) now adds PyWin32, just like the builds (the new version of pympler, a memory management module, moans on boot if it doesn't have it)

  • timestamps

  • a couple places where fixed calendar time-deltas are converted to absolute datestrings now work better over longer times. going back (5 years, 3 months) should now work out the actual calendar dates (previously they used a rough total_num_seconds estimation) and go back to the same day of the destination month, also accounting for if that has fewer days than the starting month and handling leap years. it also handles >'12 months' better now
  • in system:time predicates that use since/before a delta, it now allows much larger values in the UI, like '72 months', and it won't merge those into the larger values in the label. so if you set a gap of 100 days, it'll say that, not 3 months 10 days or whatever
  • the main copy button on 'manage file times' is now a menu button letting you choose to copy all timestamps or just those for the file services. as a hacky experiment, you can also copy the file service timestamps plus one second (in case you want to try finick-ily going through a handful of files to force a certain import sort order)
  • the system predicate time parsing is now more flexible. for archived, modified, last viewed, and imported time, you can now generally say all variants in the form 'import' or 'imported' and 'time' or 'date' and 'time imported' or 'imported time'.
  • fixed an issue that meant editing existing delta 'system:archived time' predicates was launching the 'date' edit panel

  • misc

  • in the 'exif and other embedded metadata' review window, which is launched from a button on the the media viewer's top hover, jpegs now state their subsampling and whether they are progressive
  • every simple place where the client eats clipboard data and tries to import something now has a unified error-reporting process. before, it would make a popup with something like 'I could not understandwhat was in the clipboard!'. Now it makes a popup with info on what was pasted, what was expected, and actual exception info. Longer info is printed to the log
  • many places across the program say the specific exception type when they report errors now, not just the string summary
  • the sankaku downloader is updated with a new url class for their new md5 links. also, the file parser is updated to associate the old id URL, and the gallery parser is updated to skip the 'get sank pro' thumbnail links if you are not logged in. if you have sank subscriptions, they are going to go crazy this week due to the URL format changing--sorry, there's no nice way around it!--just ignore their popups about hitting file limits and wait them out. unfortunately, due to an unusual 404-based redirect, the id-based URLs will not work in hydrus any more
  • the 'API URL' system for url classes now supports File URLs--this may help you figure out some CDN redirects and similar. in a special rule for these File URLs, both URLs will be associated with the imported file (normally, Post API URLs are not saved as Known URLs). relatedly, I have renamed this system broadly to 'api/redirect url', since we use it for a bunch of non-API stuff now
  • fixed a problem where deleting one of the new inc/dec rating services was not clearing the actual number ratings for that service from the database, causing service-id error hell on loading files with those orphaned rating records. sorry for the trouble, this slipped through testing! any users who were affected by this will also be fixed (orphan records cleared out) on update (issue #1357)
  • the client cleans up the temporary paths used by file imports more carefully now: it tries more times to delete 'sticky' temp files; it tries to clear them again immediately on shutdown; and it stores them all in the hydrus temp subdirectory where they are less loose and will be captured by the final directory clear on shutdown (issue #1356)

• version 524

  • timestamp sidecars

  • the sidecars system now supports timestamps. it just uses the unix timestamp number, but if you need it, you can use string conversion to create a full datestring. each sidecar node only selects/sets that one timestamp, so this may get spammy if you want to migrate everything, but you can now migrate archived/imported/whatever time from one client to another! the content updates from sidecar imports apply immediately _after_ the file is fully imported, so it is safe and good to sidecar-import 'my files imported time' etc.. for new files, and it should all get set correctly, but obviously let me know otherwise. if you set 'archived time', the files have to be in an archived state immediately after import, which means importing and archiving them previously, or hitting 'archive all imports' on the respective file import options
  • sidecars are getting complex, so I expect I will soon add a button that sets up a 'full' JSON sidecar import/export in one click, basically just spamming/sucking everything the sidecar system can do, pretty soon, so it is easier to set up larger migrations

  • timestamp merge

  • the duplicate merge options now have an action for 'sync file modified date?'. you can set so both files get their earliest (the new default for 'they are the same'), or that the earlier worse can be applied to the later better (the new default for 'this is better') (issue #1203)
  • in the duplicate system, when URLs are merged, their respective domain-based timestamps are also merged according to the earliest, as above

  • more timestamps

  • hydrus now supports timestamps before 1970. should be good now, lol, back to 1AD (and my tests show BC dates seem to be working too?). it is probably a meme to apply a modified date of 1505 to some painting, but when I add timestamps to the API maybe we can have some fun. btw calendar calculations and timezones are hell on earth at times, and there's a decent chance that your pre-1970 dates may show up on hour out of phase in labels (a daylight savings time thing) of what you enter in some other area of UI. in either case, my code is not clever enough to apply DST schedules retroactively to older dates, so your search ranges may simply be an hour out back in 1953. it sounds stupid, but it may matter if we are talking midnight boundaries, so let me know how you find it
  • when you set a new file modified date, the file on disk's modified date will only be updated if the date set is after 1980-01-01 (Windows) or 1970-01-01 (Linux) due to system limitations
  • fixed a typo bug in last week's work that meant file service timestamp editing was not updating the media object (i.e. changes were not visible until a restart)
  • fixed a bug where collections that contained files with delete timestamps were throwing errors on display. (they were calculating aggregate timestamp data wrong)
  • I rejiggered how the 'is this timestamp sensible?' test applies. this test essentially discounts any timestamp before 1970-01-08 to catch any weird mis-parses and stop them nuking your aggregate modified timestamp values. it now won't apply to internal duplicate merge and so on, but it still applies when you parse timestamps in the downloader system, so you still can't parse anything pre-1970 for now
  • one thing I noticed is my '5 years 1 months ago' calculation, which uses a fixed 30 day month and doesn't count the extra day of leap years, is showing obviously increasingly inaccurate numbers here. I'll fix it up

  • export folders

  • export folders can now show a popup while they work. there's a new checkbox for it in their edit UI. default is ON, so you'll start seeing popups for export folders that run in the background. this popup is cancellable, too, so you can now stop in-progress export runs if things seem wrong
  • both import and export folders will force-show working popups whenever you trigger them manually
  • export folders no longer have the weird and confusing 'paused' and 'run regularly?' duality. this was a legacy error handling thing, now cleaned up and merged into 'run regularly?'
  • when 'run regularly?' is unchecked, the run period and new 'show popup while working regularly?' checkboxes are now disabled

  • misc

  • added 'system:ratio is square/portrait/landscape' nicer label aliases for =/taller/wider 1:1 ratio. I added them to the quick-select list on the edit panel, too. they also parse in the system predicate parser!
  • I added a bit to the 'getting started with downloading' help page about getting access to difficult sites. I refer to Hydrus Companion as a good internal login solution, and link to yt-dlp, gallery-dl, and imgbrd-grabber with a little discussion on setting up external import workflows. I tried gallery-dl on twitter this week and it was excellent. it can also take your login credentials as either user/pass or cookies.txt (or pull cookies straight from firefox/safari) and give access to nsfw. since twitter has rapidly become a pain for us recently, I will be pointing people to gallery-dl for now
  • fixed my Qt subclass definitions for PySide6 6.5.0, which strictly requires the Qt object to be the rightmost base class in multiple inheritance subclasses, wew. this his AUR users last week, I understand!

  • client api (and local booru lol)

  • if you set the Client API to not allow non-local connections, it now binds to 127.0.0.1 and ::1 specifically, which tell your OS we only want the loopback interface. this increases security, and on Windows _should_ mean it only does that first-time firewall dialog popup when 'allow non-local connections' is unchecked
  • I brushed up the manage services UI for the Client API. the widgets all line up better now, and turning the service on and off isn't the awkward '[] do not run the service' any more
  • fixed the 'disable idle mode if the client api does stuff' check, which was wired up wrong! also, the reset here now fires as a request starts, not when it is complete, meaning if you are already in idle mode, a client api request will now quickly cancel idle mode and hopefully free up any locked database situation promptly

  • boring cleanup and stuff

  • reworked all timestamp-datetime conversion to be happier with pre-1970 dates regardless of system/python support. it is broadly improved all around
  • refactored all of the HydrusData time functions and much of ClientTime to a new HydrusTime module
  • refactored the ClientData time stuff to ClientTime
  • refactored some thread/process functions from HydrusData to HydrusThreading
  • refactored some list splitting/throttling functions from HydrusData to a new HydrusLists module
  • refactored the file filter out of ClientMedia and into the new ClientMediaFileFilter, and reworked things so the medialist filter jobs now happen at the filter level. this was probably done the wrong way around, but oh well
  • expanded the new TimestampData object a bit, it can now give a nice descriptive string of itself
  • wrote a new widget to edit TimestampData stubs
  • wrote some unit tests for the new timestamp sidecar importer and exporter
  • updated my multi-column list system to handle the deprecation of a column definition (today it was the 'paused' column in manage export folders list)
  • it should also be able to handle new column definitions appearing
  • fixed an error popup that still said 'run repair invalid tags' instead of 'run fix invalid tags'
  • the FILE_SERVICES constant now holds the 'all deleted files' virtual domain. this domain keeps slipping my logic, so fingers crossed this helps. also means you can select it in 'system:file service' and stuff now
  • misc cleaning and linting work

• version 523

  • timestamp editing

  • you can now _right-click->manage->times_ on any file to edit its archived, imported, deleted, previously imported (for undelete), file modified, domain modified, and last viewed times. there's a whole new dialog with new datetime buttons and everything. it only works on single files atm, so it is currently only appropriate for little fixes, and there's a couple advanced things like setting a currently missing deletion time that it can't do yet, but I expect to expand it in future (also ideally with some kind of 'cascade' option for multi-files so you can set a timestamp iteratively (e.g. +1 second per file) over a series of thumbs to force a certain import order sort etc...)
  • I added a new shortcut action 'manage file times', for this dialog. like the other media 'manage' shortcuts, you can hit it on the dialog to ok it, too
  • when you edit a saved file modified date, I have made it to update the actual file modified date on your disk too. a statement is printed to the log with old/new timestamps, just in case you ever need to recover this
  • added system:archived time search predicate! it is under the system:time stub like the other time-based search preds. it works in the system predicate parser too

  • misc

  • fixed a stupid logical typo from 521's overhaul that was causing the advanced file deletion dialog to always set the default file deletion reason! sorry for the trouble, this one slipped through due to a tricky test situation (this data is actually calculated twice on dialog ok, and on the first run it was correct -\_-)
  • in the edit system predicate dialogs, when you have a list of 'recent' preds and static useful preds, if one of the recent is supposed to also appear in the statics, it now won't be duped
  • fixed a bug in the media object's file locations manager's deletion routine, which wasn't adding and removing the special 'all deleted files' domain at the UI level--not that this shows up in UI much, but the new timestamps UI revealed this
  • in the janitorial 'petitions processing' page, the add and delete checkbox lists now no longer have horizontal scrollbars in any situation. previously, either list, but particularly the 'delete', at height 1, could be deceptively obscured by a pop-in scrollbar
  • when you change your internal backup location, the dialog now states your current location beforehand. this information was previously not viewable! also, if you select the same location again, the process notes this and exits with no changes made
  • all multi-column lists across the program now show a ▲ or ▼ on the column they are currently sorted on! this is one of those things I meant to do for ages; now it is done.
  • also, you can now right-click any multi-column list's header for a stub menu. for now it just says the thing's identifier name, but I'll start hanging things off here like individual section-size reset and, in time, finally play around with 'select columns' tech
  • all menus across the program now send their longer description text to the main window status bar. until now (at least in Qt, I forget wx), this has only been true for the menubar menus
  • all menus across the program now have tooltips turned on. any command with description text, which is I think pretty much all of them, will present its full written description on hover. this may end up being annoying, so let me know what you think

  • client api

  • fixed an issue in the client api where it wasn't returning `file_metadata` results in the same file order you asked for. sorry for the trouble--this was softly intended, previously, but I forgot to make sure it stayed true. it also now folds in 'missing' hashes with null ids in the same position you asked for
  • a new suite of unit tests check this explicitly for all the typical parameter/response types, and the new missing-hash insertion order--it shouldn't happen again!
  • just to be safe, since this is a new feature, client api version is now 44

  • boring code updates/cleanup

  • wrote a new serialisable 'timestamp data' object to hold the various hydrus timestamps: archived, imported, deleted, previously imported, file modified, domain modified, aggregate modified, and last viewed time
  • rewrote the timestamp content update pipeline to use 'timestamp data' object
  • wrote a new database module for timestamp management off the file metadata module and migrated the domain-based modified timestamp code to it
  • migrated the 'archive time' timestamp-handling from the inbox module to the new timestamp module
  • migrated the media result timestamp-manager construction routine all down to the new timestamp module
  • migrated the aggregate modified time file search code to the new timestamp module and added archived time search too
  • wrote some UI for timestamp editing, whacked some copy/paste buttons on it too
  • moved all current/deleted timestamp handling down from the locations manager to the timestamp manager and split off 'previously imported' time, which is used to preserve import timestamp for undelete events, into its own thing rather than a tacked-on hack for deleted timestamps
  • moved all the location manager location timestamp tracking down to the timestamp manager
  • the media result is now initialised with and handles an explicit copy of the timestamp manager, which is now shared to both location manager and file viewing stats manager, with duplication and merging code updated to handle this shared situation
  • moved all the media/preview 'last view time' tracking down from the file viewing stats manager to the timestamp manager, which FVS now received on initialisation
  • all media-based timestamp inspection now goes through the timestamp manager
  • collections now track some aggregate timestamps a bit better, and they now calculate a archived time--not sure if it is useful, but they know it now
  • updated all parts of the timestamp system to use the same shared enums
  • cleaned the timestamp code generally
  • cleaned some file service update code generally
  • moved the main file viewing stats fetching routine for MediaResult building down to the file viewing stats module
  • updated the old custom gridbox layout to handle multiple-column-spanning controls
  • went through all the bash scripts and fixed some issues my IDE linter was moaning about. -r on reads, quotes around variable names, 4-space indenting, and neater testing of program return states

• version 522

  • notes in sidecars

  • the sidecars system now supports notes!
  • my sidecars only support univariate rows atm (a list of strings, rather than, say, a list of pairs of strings), so I had to make a decision how to handle note names. if I reworked the pipeline to handle multivariate data, it would take weeks; if I incorporated explicit names into the sidecar object, it would have made 'get/export all my notes' awkward or impossible and not solved the storage problem; so I have compromised in this first version by choosing to import/export everything and merging the name and text into the same row. it expects/says 'name: text' for input and output. let me know what you think. I may revisit this, depending on how it goes
  • I added a note to the sidecars help about this special 'name: text' rule along with a couple ideas for tricky situations

  • misc

  • added 'system:framerate' and 'system:number of frames' to the system predicate parser!
  • I am undoing two changes to tag logic from last week: you can now have as many colons at the start of a tag as you like, and the content parser no longer tries to stop double-stacked namespaces. both of these were more trouble than they were worth. in related news, '::' is now a valid tag again, displaying as ':', and you can create ':blush:'-style tags by typing '::blush:'. I'm pretty sure these tags will autocomplete search awfully, so if you end up using something like this legit, let me know how it goes
  • if you change the 'media/preview viewer uses its own volume' setting, the client now updates the UI sliders for this immediately, it doesn't need a client restart. the actual volume on the video also changes immediately
  • when an mpv window is called to play media that has 'no audio', the mpv window is now explicitly muted. we'll see if this fixes an interesting issue where on one system, videos that have an audio channel with no sound, which hydrus detects as 'no audio', were causing cracks and pops and bursts of hellnoise in mpv (we suspect some sort of normalisation gain error)

  • file safety with duplicate symlinked directory entries

  • the main hydrus function that merges/mirrors files and directories now checks if the source and destination are the same location but with two different representations (e.g. a mapped drive and its network location). if so, to act as a final safety backstop, the mirror skips work and the merge throws an error. previously, if you wangled two entries for the same location into 'migrate database' and started a migration, it could cause file deletions!
  • I've also updated my database migration routines to recognise and handle this situation explicitly. it now skips all file operations and just updates the location record instantly. it is now safe to have the same location twice in the dialog using different names, and to migrate from one to the other. the only bizzaro thing is if you look in the directory, it of course has boths' contents. as always though, I'll say make backups regularly, and sync them before you do any big changes like a migration--then if something goes wrong, you always have an up-to-date backup to roll back to
  • the 'migrate database' dialog no longer chases the real path of what you give it. if you want to give it the mapped drive Z:, it'll take and remember it
  • some related 'this is in the wrong place' recovery code handles these symlink situations better as well

  • advanced new parsing tricks

  • thanks to a clever user doing the heavy lifting, there are two neat but advanced additions to the downloader system
  • first, the parsing system has a new content parser type, 'http headers', which lets you parse http headers to be used on subsequent downloads created by the parsing downloader object (e.g. next gallery page urls, file downloads from post pages, multi-file posts that split off to single post page urls). should be possible to wangle tokenized gallery searches and file downloads and some hacky login systems
  • second, the string converter system now lets you calculate the normal hydrus hashes--md5, sha1, sha256, sha512--of any string (decoding it by utf-8), outputting hexadecimal

  • http headers on the client api

  • the client api now lets you see and edit the http headers (as under _network->data->review http headers_) for the global network context and specific domains. the commands are `/manage_headers/get_headers` and `/manage_headers/set_headers`
  • if you have the 'Make a short-lived popup on cookie updates through the Client API' option set (under 'popups' options page), this now applies to these header changes too
  • also debuting on the side is a 'network context' object in the `get_headers` response, confirming the domain you set for. this is an internal object that does domain location stuff all over. it isn't important here, but as we do more network domain setting editing, I expect we'll see more of this guy
  • I added some some documentation for all this, as normal, to the client api help
  • the labels and help around 'manage cookies' permission are now 'manage cookies and headers'
  • the client api version is now 43
  • the old `/manage_headers/set_user_agent` still works. ideally, please move to `set_headers`, since it isn't that complex, but no rush. I've made a job to delete it in a year
  • while I was doing this, I realised get/set_cookies is pretty bad. I hate their old 'just spam tuples' approach. I've slowly been replacing this stuff with nicer named JSON Objects as is more typical in APIs and is easier to update, so I expect I'll overhaul them at some point

  • boring cleanup

  • gave the about window a pass. it now runs on the newer scrolling panel system using my hydrus UI objects (so e.g. the hyperlink now opens on a custom browser command, if you need it), says what platform you are on and whether you are source/build/app, and the version info lines are cleaned a little
  • fixed/cleaned some bad code all around http header management
  • wrote some unit tests for http headers in the client api
  • wrote some unit tests for notes in sidecars

• version 521

  • some tag presentation

  • building on last week's custom sibling connector, if you don't like the fade you can now override the 'namespace' colour of the sibling connector if you like
  • you can also set the ' OR ' connector text
  • and you can set the OR connector's 'namespace' colour. it was 'system' before
  • also turned off the new namespace colour fading for OR predicates, where it was unintentionally kicking in and looking horrible lol

  • misc

  • added a checkbox to 'file viewing statistcs' to turn off tracking for the archive/delete filter, if you don't like that
  • file viewing statistics now maxes out at five times a duration-having media's duration, if that is more than your max view time
  • the simple version of the file delete dialog will now never overwrite a file deletion reason if all of the to-be-deleted files already have deletion reasons (e.g. when physically deleting trash)
  • the advanced version of the dialog now always selects 'keep existing reason' or 'do not alter existing reasons' when they exist, regardless of your 'remember previous reason' action. also, the 'remember previous reason' saved reason no longer updates if 'keep existing reason' or 'do not alter existing reasons' is set--it will stick on whatever it was before
  • I might have fixed a height-layout bug in the petition management page

  • advanced change to unnamespaced tags and their parsing

  • the rule that allows ':p' as a tag (by secretly storing it as '::p') has been expanded--now any unnamespaced tag can include a colon as long as it starts with an explicit colon, which in hydrus rendering contexts is usually hidden. you can now type these in simply by beginning your tag with ':'--the secret character will be quickly swallowed
  • for the parsing system, content parsers that get tags can now decide whether to set an explicit namespace or not. from now on, content parsers that are set to get unnamespaced tags will force all tags they get to be unnamespaced! this stops some site that has incidental colons in their 'subtags' from spamming twenty different new namespaces to hydrus. to preserve old parser behaviour, all existing content parsers that were left blank (no namespace) will be updated to not set an explicit namespace. if you are a parser maker, please consider whether you want to go with 'unnamespaced' or 'any namespace' going forward in your parsers--since most places don't use the hydrus 'namespace:subtag' format, I suspect when we want to make the decision, we'll want 'unnamespaced'
  • I updated the pixiv parser to specifically ask for unnamespaced tags when parsing regular user tags, since it has some of these colon-having tags
  • as a side thing, extra colons are now collapsed at the start of a tag--anything that starts with four colons will be collapsed down to two, with one displaying to humans
  • also, during parsing, if a content parser gets a tag and the subtag already starts with its namespace, it will no longer double the namespace. parse 'character:dave' with namespace set to 'character', it will no longer produce 'character:character:dave'

  • advanced file domain and file import options stuff

  • all import pages that need to consult their file domain now do so on a 'realised' version of 'default file import options', so if you are set to import to 'my imports', and you open a new page from a tag or some thumbs on that import page, the new file page will be set to 'my imports', not some weird 'my files' stub value (in clients that deleted 'my files', this would be 'initialising...' forever)
  • more stages of the file import process 'realise' default file import options stubs, just in case more of these problems slip through in future (e.g. in my file import unit tests, which I just discovered were all broken)
  • the 'default' file import options stub is now initialised with your first local file domain rather than 'my files', so if this thing is ever still consulted anywhere, it should serve as a better last resort
  • also fixed the file domain button getting stuck on 'initialising' if it starts with an empty file domain
  • when you open the edit file import options dialog on a 'default' FIO and switch to a non-default, it now fills in all the details with the current LOUD FIO

  • boring cleanup

  • extracted the master file search method (~1800 lines of code) from the monolithic database object and into its own module. then broke several sub-pieces like rating or note searching code out into that module and cleaned misc stuff along the way. not done by any means, but this was a big db-cleanup hump
  • reshuffled all the page management objects so they no longer keep an explicit copy of their current file domain--now they always consult their respective sub-objects, whether that is a file search or an importer or what. any time a page needs to consult its file domain, it'll always get the live and sensible version. as above, they also 'realise' default file import options stubs
  • broke the 'getting started with tags' help page into two and straddled the 'getting started with searching' page with them. the intention is to get users typing a few tags into their first import pages, just that, and then playing around with them in search, before moving on to more complicated tag subjects
  • split the 'autocomplete' section of the 'search' options into two, for read/write a/c contexts, and the default file and tag domain options have been moved there from 'files and trash' and 'tags'

• version 520

  • autocomplete

  • in autocomplete dropdowns, the advanced 'all known files' file domain now generally appears as 'all known files with tags'. the way file+tag search works here has been obscure and confusing for a long time; now the label specifically says what's going on
  • to complement 'all known files with tags', all users now see a new 'all files ever imported/deleted', which is what most people actually want when they try 'all known files'. this quick-select entry for 'currently in or deleted from all my files' will run super quick in almost all cases and allows 'all known tags'!
  • the new 'preserve selection between prefetch and full results' behaviour in tag autocomplete no longer applies if you have 'select the first item with count' turned on. these things just don't play well together
  • that 'select the first item with count' option is now available in the manage tags dialog's cog icon too
  • the 'edit' autocomplete tag search should be better about shuffling the top results. it now tries to put 'ideal of what you entered' at the very top (if that differs from what you typed), then what you actually typed (with or without count), and no longer shuffles other siblings to the top--while they are still included in the results, they weren't so helpful being spammed to the top every time!
  • any search predicate that has a wildcard asterisk in its namespace is now coloured by default as the 'namespaced tags' fallback colour. this includes the somewhat new (any namespace) search tags. behind the scenes, the colour I assign is for a namespace of just '*', so you can set your own colour if you like
  • the different 'edit tags' autocomplete panels that have paste buttons--in manage siblings, manage parents, filename tagging, tag import options, and favourite tag management--are now all 'add only'. if any of the tags you are pasting already exist in the list, they now won't be removed

  • misc

  • the '(displays as xxx)' sibling suffix is shortened to a simpler unicode arrow, " → ". if you don't like it, you can edit it under _options->tag presentation_!
  • I also went full meme and made the sibling connecting block's background colour a gradient on Qt6 (and lol the unselected text is a gradient too, but you need to alter it to something longer to really see). if you don't like it, you can turn it off in the same place! I also tweaked some of the padding sizes here so the different text blocks line up a little nicer
  • thanks to a user's continued good work, I am rolling in another update to the Deviant Art file downloader that can grab the 'original quality' file from the logged-in-only download button that some artists turn on. furthermore, there are five new 'File URL' classes for the different qualities the file urls represent, which will propagate to all of your existing DA files, be searchable with system:known url, and hence allow you to find the medium/original/whatever quality versions that you have. now, not every 'medium quality' post on the site has the 'original' download button, but if you are an advanced user with a long DA download history, then with a bit of magic wand waving with your file import options, you can set up an url downloader for a one-time rescan that'll check and redownload your favourite mediums' URLs, or the mediums you know will have 'original quality', for that better version--try it in a small batch first, and let me know what you discover!
  • fixed note content update pipeline so it can handle various instances of multiple notes with the same name coming in at once. previously it would pseudorandomly pick one and discard the others, now it does all the normal '(1)' renaming rules (and even note text extension merging, and hopefully in a good reliable order) as it goes through them
  • if you are a madlad, you can now boost the 'prefetch previous/next' options under 'speed and memory' up to 50 either way. a new label complains if you set them too high given your current image cache size
  • the file maintenance system now catches serious IOErrors, which usually suggest big deal hard drive problems, give the user a special popup message, and stops all future file maintenance work that boot
  • the file maintenance system is better at stopping work for program shutdown while in the midst of a larger batch job
  • fixed the second 'current and pending' label on 'migrate tags'--the new action was 'pending only' as intended, the bad label was just a stupid copy/paste typo
  • thanks to a detailed user report, fixed multiple broken internal #anchor links in the help

  • repository

  • (both server and client need to be updated to get this)
  • last week's 'delete all content' command failed IRL. it locked up the PTR for six hours and then appeared to fail (rollback) on a seemingly normal account. I am not sure what the inefficiency was here, but this job obviously has to be re-thought for real world use, so this week I am altering the command to break the job up into smaller pieces and stop safely after twenty seconds of work. the janitor client will receive a message on whether everything was deleted or not
  • this is not a total solution or a nice solution, but it should be a stopgap that still allows deletion of small accounts' content while not breaking for big accounts. the ultimate answer here is going to look like proper account content-count caching (rather than the '5000 mappings' limit), and an asynchronous 'purge' maintenance system that runs in the background that janitor clients can check up on and even cancel

• version 519

  • inc/dec ratings service

  • I have written a new number 'rating' service type, called 'inc/dec'. it is simply a no-upper-limit positive integer--you left-click to increment, right to decrement. middle-click to edit directly
  • it appears and works like other ratings in the top-right media viewer hover and the manage ratings dialog. there's a section under system:ratings too. the main logical difference is every file is always rated in this system--the default for all files is 0--so there's no searching for 'unrated'
  • the duplicate merge options support this new inc/dec rating by adding/summing in one or both directions. its action labels in the dialog are a little different because of this

  • misc

  • the manage tag siblings dialog now shows all members of a chain when it filters the current in-view pairs according to the current pertinent tags. previously, it just showed the pairs that included your entered tags; now it chases everything
  • the same is also now broadly true of manage tag parents, but there's a checkbox that sets how crazy it goes. by default it won't pursue 'cousins', since that can make a really overwhelming list (imagine seeing every character nintendo ever created, including every pokemon, when you just wanted to add a samus costume variant). more work can and will be done here, also with sibling-cross referencing
  • the system:ratings panel now lists the groups of rating services in alphabetical order
  • fixed an issue where the hydrus native animation renderer was drawing animations at small size in the top-left with garbled surrounds when the monitor UI scale was >100% (issue #1334)
  • I think I have hacked an ugly fix for the 'this window keeps growing horizontally until it reaches the width of the screen' bug that hits some people. the sizing code is now supposed to recognise when this happens and stop it in place. if you get this problem, let me know if it is fixed or what! (issue #1331)
  • if a file in the duplicate filter (or any other media viewer, if you can wangle it) has a 'show action' of 'do not show in the media viewer' or 'do not show, open externally on thumbnail activate', the media viewer now falls back to 'show open externally button'. previously, it was halting in an ugly state and no longer able to proceed (issue #1329)
  • if repository processing runs into any missing/invalid file trouble, it now queues up a wider array of potential file maintenance jobs, assuming there may be a problem with the file records themselves
  • if, during repository processing, an update file is missing, the error note now asks users to run _database->maintenance->clear orphan file records_. might be that the above fix helps here too, but this will be the sledgehammer solution on top, clearing up unusual cases where one service thinks the files exist when actually they don't
  • fixed the recent 'when ffmpeg can't generate a video thumb, use hydrus thumb' routine to cover more situations
  • thanks to a user, fixed a bunch of unit tests for python 3.11

  • misc cleanup

  • updated my async updater object to handle some pre-call UI-side argument-construction and cleaned up some related garbage shared memory hacks I had before
  • in a step towards less laggy sibling/parents dialogs, I have moved the 'manage tag siblings' dialog's list-filtering routine to a thread. I'll do parents too, sometime, and plan to eventually move to very fast on-demand existing-pair fetching based on the above lookup rule improvements rather than the super laggy 'load everything on dialog boot' current system. a next big step would obviously be visual graph representation of sibling and parent chains
  • cleaned some ratings code and fixed some weird little bugs like numerical rating tooltips not updating properly after a click
  • added some unit tests for inc/dec ratings

  • server admin

  • (the server and client both need to be updated to get this)
  • I updated and reinstated the old 'superban' function for janitors! it is now just 'delete all account content' on the account modification dialog, separate from the banning process. note that since the server only remembers account ownership of content through the anonymisation period, it cannot auto-remove content older than that date!
  • the account info you see in the modify account dialog now only shows file count/bytes for file repositories and tag counts for tag repositories. to improve readability, it also shows every key/value pair on a separate line, sorted by keys
  • that account info now shows, for tag repositories, number of current, pending, and petitioned sibling and parent rows, and it shows number of petitioned mapping rows. all this stuff obviously goes to 0 if you hit 'delete all account content'--let me know if any of it doesn't!
  • the modify accounts dialog no longer shows the 'null' account type as a choice to set things to. duh! its yes/no also now confirms the account type you are settting
  • all the commands in the modify accounts dialog now have nicer yes/no dialogs that say the number of accounts being affected and talk more about what is happening
  • fixed up some logical jank in the dialog. adding time to expires no longer tells you about 0 accounts having no expiry, and if circumstances mean 0 accounts are selected/valid for an operation, it no longer says 'want to set expiry for 0 accounts?' etc...
  • when modifying multiple accounts, the current account focus/selection is now preserved through list refreshes after jobs go through

• version 518

  • autocomplete improvements

  • tl;dr: I went through the whole tag autocomplete search pipeline, cleaned out the cruft, and made the pre-fetch results more sensible. searching for tags on thumbnails isn't horrible any more!
  • -
  • when you type a tag search, either in search or edit autocomplete contexts, and it needs to spend some time reading from the database, the search now always does the 'exact match' search first on what you typed. if you type in 'cat', it will show 'cat' and 'species:cat' and 'character:cat' and anything else that matches 'cat' exactly, with counts, and easy to select, while you are waiting for the full autocomplete results to come back
  • in edit contexts, this exact-matching pre-fetch results here now include sibling suggestions, even if the results have no count
  • in edit contexts, the full results should more reliably include sibling suggestions, including those with no count. in some situations ('all known tags'), there may be too many siblings, so let me know!
  • the main predicate sorting method now sorts by string secondarily, stabilising the sort between same-count preds
  • when the results list transitions from pre-fetch results to full results, your current selection is now preserved!!! selecting and then hitting enter right when the full results come in should be safe now!
  • when you type on a set of full results and it quickly filters down on the results cache to a smaller result, it now preserves selection. I'm not sure how totally useful this will be, but I did it anyway. hitting backspace and filtering 'up' will reset selection
  • when you search for tags on a page of thumbnails, you should now get some early results super fast! these results are lacking sibling data and will be replaced with the better answer soon after, but if you want something simple, they'll work! no more waiting ages for anything on thumbnail tag searches!
  • fixed an issue where the edit autocomplete was not caching results properly when you had the 'unnamespaced input gives (any namespace) wildcard results' option on
  • the different loading states of autocomplete all now have clear 'loading...' labels, and each label is a little different based on what it is doing, like 'loading sibling data...'
  • I generally cleared out jank. as the results move from one type to another, or as they filter down as you type, they _should_ flicker less
  • added a new gui debug mode to force a three second delay on all autocomplete database jobs, to help simulate slow searches and play with the above
  • NOTE: autocomplete has a heap of weird options under _tags->manage tag display and search_. I'm really happy with the above changes, but I messed around with the result injection rules, so I may have broken one of the combinations of wildcard rules here. let me know how you get on and I'll fix anything that I busted.

  • pympler

  • hydrus now optionally uses 'pympler', a python memory profiling library. for now, it replaces my old python gc (garbage collection) summarising commands under _help->debug->memory actions_, and gives much nicer formatting and now various estimates of actual memory use. this is a first version that mostly just replicates old behaviour, but I added a 'spam a more accurate total mem size of all the Qt widgets' in there too. I will keep developing this in future. we should be able to track some memory leaks better in future
  • pympler is now in all the requirements.txts, so if you run from source and want to play with it, please reinstall your venv and you'll be sorted. _help->about_ says whether you have it or not

  • misc

  • the system:time predicates now allow you to specify the hh:mm time on the calendar control. if needed, you can now easily search for files viewed between 10pm-11:30pm yesterday. all existing 'date' system predicates will update to midnight. if you are a time-search nerd, note this changes the precision of existing time predicates--previously they searched _before/after_ the given date, but now they search including the given date, pivoting around the minute (default: 0:00am) rather than the integer calendar day! 'same day as' remains the same, though--midnight to midnight of the given calendar day
  • if hydrus has previously initial-booted without mpv available and so set the media view options for video/animations/audio to 'show with native viewer', and you then boot with mpv available, hydrus now sets your view options to use mpv and gives a popup saying so. trying to get mpv to work should be a bit easier to test now, since it'll popup and fix itself as soon as you get it working, and people who never realised it was missing and fix it accidentally will now get sorted without having to do anything extra
  • made some small speed and memory optimisations to content processing for busy clients with large sessions, particularly those with large collect-by'd pages
  • also boosted the speed of the content update pipeline as it consults which files are affected by which update object
  • the migrate tags dialog now lets you filter the tag source by pending only on tag repositories
  • cleaned up some calendar/time code
  • updated the Client API help on how Hydrus-Client-API-Access-Key works in GET vs POST arguments
  • patched the legacy use of 'service_names_to_tags' in `/add_urls/add_url` in the client api. this parameter is more obsolete than the other legacy names (it got renamed a while ago to 'service_names_to_additional_tags'), but I'm supporting it again, just for a bit, for Hydrus Companion users stuck on an older version. sorry for the trouble here, this missed my legacy checks!

  • windows mpv test

  • hey, if you are an advanced windows user and want to run a test for me, please rename your mpv-2.dll to .old and then get this https://sourceforge.net/projects/mpv-player-windows/files/libmpv/mpv-dev-x86_64-20230212-git-a40958c.7z/download . extract the libmpv-2.dll and rename it to mpv-2.dll. does it work for you, showing api v2.1 in _help->about_? are you running the built windows release, or from source? it runs great for me from source, but I'd like to get a wider canvas before I update it for everyone. if it doesn't work, then delete the new dll and rename the .old back, and then let me know your windows version etc.., thank you!

• version 517

  • misc

  • thanks to a user, export folders finally support exporting to symlinks!
  • if a symlink export-create fails on Windows, the error now tells you to try again in 'run as Admin' mode--seems like this is needed in Win 10+ unless you mess with Group Policy Editor
  • 'related tags' should no longer suggest sibling ideals or parents of existing tags! I think!
  • when a thumbnail fails to load, the error popup now has a button to open the specific problem-causing file in a new page
  • generation of video thumbnails is faster, should fail less in odd cases, and when it completely fails, it now gives the hydrus icon as a final fallback
  • generation of image thumbnails now falls back to the hydrus icon as a final fallback
  • I think I fixed a focus logic problem where the autocomplete dropdowns on the duplicate filter page would hide if you clicked a results/favourites tab or greyspace
  • fixed an error when seeking an mpv video while the video was loading or unloading
  • the max 'nullification period' (after which uploads to a hydrus repository are anonymised) is raised from 1 year to 5 (needs server and client update to work)

  • transparency and duplicate filter

  • two new options, under _media_ and _duplicates_, now control if you would like transpararency-having images to have a checkerboard background rather than the normal media canvas background! you can have it on all the time or just under the duplicate filter. it uses the same style of grid as MPV
  • I have a plan for proper native (non-MPV) transparency for gifs and apng, but I think I'll wait for an imagemagick plugin I am planning first
  • if you have a white/black media viewer background and prefer not to use the checkerboard, the duplicate filter can now adjust the background colour, either lighter or darker, for both A and B of the pair. altering A as well exposes truly transparent-having images vs ones with opaque white/black fill, which will otherwise blend into a purely white/black background colour. these options are available in the options dialog and the duplicate filter right-hand hover window cog button
  • the native image window, embed button, and animation window (with PIL gif rendering) now all adjust their background colour to any odd changes like the duplicate filter's A/B lighten/darken adjustment

  • boring cleanup

  • cleaned up how popup file buttons are set and cleared
  • cleaned up how popup main and secondary texts are set and cleared
  • misc linting cleanup

• version 516

  • misc

  • the 'manage sidecar routers' control, which is on manage import folders, manage export folders, path-tagging-before-manual-import, and manual export files, now has import/export/duplicate buttons. you can save and transfer your work now! if you try to import 'export to sidecar' routers to an 'import from sidecar' context or _vice versa_, it should give you a nicely worded error
  • fixed the error that was raising when you turn related tags off with the suggestions set to side-by-side layout. very sorry for the trouble!
  • apngs that are set to 'loop x times' (usually once) now only loop that many times, on both mpv and my native renderer! like gifs, the 'always loop animations' setting under _options->media_ overrides it!
  • fixed an issue with my native renderer not updating on scanbar scrubs very well. should be back to nice smooth instant draw as you scrub
  • thanks to a user, folded in another deviant art parser update to the defaults
  • updated the setuptools version in the requirements.txt due to a security note--I don't think the problem (which was about some vulnerable regex when fetching malicious package info) applies to us, but running from source users might like to run setup_venv again this week anyway

  • related tags

  • a new 'concurrence threshold' setting under _options->tag suggestions_ allows you to set how 'strict' the related tags search is. a higher percentage causes fewer but more relevant results. I'm increasing the default this week from 4% to 6%
  • two new 'namespace to weight' settings under _options->tag suggestions_ now manage how much weight the 'search' and 'suggestion' sides of related tags have. you can say 'rank the suggestions from character tags highly' or 'rank unnamespaced suggestions lower', and 'do not search x tags' and 'do not suggest y tags'. I have prepped it with some 'creator/character/series namespaces are better than unnamespaced, and title/filename/page/chapter/volume are useless' defaults, but feel free to play around with it
  • the related tags algorithm takes a larger sample now, resulting in a _little_ less ranking-variability

  • client api

  • changed and fixed an issue in the client api's new `get_file_relationships` call. previously, I said 'king' would be null if it was not on the given file domain, but this was not working correctly--it was giving pseudorandom 'fallback' kings. now it always gives the king, no matter what! a new param, `king_is_on_file_domain` says whether the king is on the given domain. `king_is_local` says whether the king is available on disk
  • added some discussion and a list of the 8 possible 'better than' and 'same quality' logical combinations to the `set_file_relationships` help so you can see how group merge involving non-kings works
  • client api is now version 42

• version 515

  • related tags

  • I worked on last week's related tags algorithm test, bringing it up to usable standard. the old buttons now use the new algorithm exclusively. all users now get 'related tags' showing in manage tags by default (if you don't like it, you can turn it off under _options->tag suggestions_)
  • the new algorithm has new cancel tech and does a 'work for 600ms' kind of deal, like the old system, and the last-minute blocks from last week are gone--it will search as much as it has time for, including partial results. it also won't lag you out for thirty seconds (unless you tell it to in the options). it searches tags with low count first, so don't worry if it doesn't get to everything--'1girl' usually doesn't have a huge amount extra to offer once everything else has run
  • it also uses 'hydev actually thought about this' statistical sampling tech to work massively faster on larger-count tags at the cost of some variance in rank and the odd false positive (considered sufficiently related when it actually shouldn't meet the threshold) nearer the bottom end of the tags result list
  • rather than 'new 1' and 'new 2', there is now an on/off button for searching your local files or all known files on tag repositories. 'all known files' = great results, but very slow, which the tooltip explains
  • there's also a new status label that will tell you when it is searching and how well the search went (e.g. '12/51 tags searched fully in 459ms')
  • I also added the 'quick' search button back in, since we can now repeat searches for just selections of tags
  • I fixed a couple typos in the algorthim that were messing some results
  • I fixed some tag-selection-tracking-issues with the 'select some tags to limit related tags lookup to them' feature when you moved between different media in the same manage tags dialog
  • in the manage tags dialog, if you have the suggested tag panels 'side-to-side', they now go in named boxes
  • in the manage tags dialog, if you have suggested tag panels in a notebook, 'related tags' will only refresh its search on a media change event (including dialog initialisation) when it is the selected page. it won't lag you from the background!
  • options->tag suggestions now lets you pick which notebook'd tag suggestions page you want to show by default. this defaults to 'related'
  • I have more plans here. these related tags results are very cachable, so that's an obvious next step to speed up results, and when I have done some other long-term tag improvements elsewhere in the program, I'll be able to quickly filter out unhelpful sibling and parent suggestions. more immediately, I think we'll want some options for namespace weighting (e.g. 'series:' tags' suggestions could have higher rank than 'smile'), so we can tune things a bit

  • misc

  • the 'open externally' canvas widget, which shows any available thumbnail of the flash or psd or whatever, now sizes itself correctly and draws the thumbnail nicely if you set the new thumbnail supersampling option to >100%. if your thumbnail is the wrong size (and probably in a queue to be regenerated soon), I _think_ it'll still make the window too big/small, but it'll draw the thumbnail to fit
  • if a tag content update comes in with an invalid tag (such as could happen with sidecars recently), the client now heals better. the bad tag is corrected live in more places, and this should be propagated to the UI. if you got a warning about 'you have invalid tags in view' recently but running the routine found no problems, please reboot, and I think you'll be fixed. I'm pretty sure the database wasn't being damaged at all here (it has cleaning safeguards, so it _shouldn't_ be possible to actually save bad tags)--it was just a thing to do with the UI not being told of the cleaned tag, and it shouldn't happen again. thank you for the reports! (issue #1324)
  • export folders and the file maintenance dialog no longer apply the implicit system:limit (defaults to max 10k files) to their searches!
  • old OR predicates that you load with saved searches and similar should now always have alphebetised components, and if you double-click them to remove them, they will now clear correctly (previously, they were doing something similar to the recent filetype problem, where instead of recognising themselves and deleting, they would instead duplicate a normalised (sorted) copy of themselves)
  • thanks to a user, updated the recently note-and-ai-updated pixiv parser again to grab the canonical pixiv URL and translated tags, if present
  • thanks to a user, updated the sankaku parser to grab some more tags
  • the file location context and tag context buttons under tag autocompletes now put menu separators between each type of file/tag service in their menus. for basic users, this'll be a separator for every row, but for advanced users with multiple local domains, it will help categorise the list a bit

• version 514

  • downloaders

  • twitter took down the API we were using, breaking all our nice twitter downloaders! argh!
  • a user has figured out a basic new downloader that grabs the tweets amongst the first twenty tweets-and-retweets of an account. yes, only the first twenty max, and usually fewer. because this is a big change, the client will ask about it when you update. if you have some complicated situation where you are working on the old default twitter downloaders and don't want them deleted, you can select 'no' on the dialog it throws up, but everyone else wants to say 'yes'. then check your twitter subs: make sure they moved to the new downloader, and you probably want to make them check more frequently too.
  • given the rate of changes at twitter, I think we can expect more changes and blocks in future. I don't know whether nitter will be viable alternative, so if the artists you like end up on a nice simple booru _anywhere_, I strongly recommend just moving there. twitter appears to be explicitly moving to non-third-party-friendly
  • thanks to a user's work, the 'danbooru - get webm ugoira' parser is fixed!
  • thanks to a user's work, the deviant art parser is updated to get the highest res image in more situations!
  • thanks to a user's work, the pixiv downloader now gets the artist note, in japanese (and translated, if there is one), and a 'medium:ai generated' tag!

  • sidecars

  • I wrote some sidecar help here! https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html
  • when the client parses files for import, the 'does this look like a sidecar?' test now also checks that the base component of the base filename (e.g. 'Image123' from 'Image123.jpg.txt') actually appears in the list of non-txt/json/xml ext files. a random yo.txt file out of nowhere will now be inspected in case it is secretly a jpeg again, for good or ill
  • when you drop some files on the client, the number of files skipped because they looked like sidecars is now stated in the status label
  • fixed a typo bug that meant tags imported from sidecars were not being properly cleaned, despite preview appearance otherwise, for instance ':)', which in hydrus needs to be secretly stored as '::)' was being imported as ')'
  • as a special case, tags that in hydrus are secretly '::)' will be converted to ':)' on export to sidecar too, the inverse of the above problem. there may be some other tag cleaning quirks to undo here, so let me know what you run into

  • related tags overhaul

  • the 'related tags' suggestion system, turned on under _options->tag suggestions_, has several changes, including some prototype tech I'd love feedback on
  • first off, there are two new search buttons, 'new 1' and 'new 2' ('2' is available on repositories only).. these use an upgraded statistical search and scoring system that a user worked on and sent in. I have butchered his specific namespace searching system to something more general/flexible and easy for me to maintain, but it works better and more comprehensibly than my old method! give it a go and let me know how each button does--the first one will be fast but less useful on the PTR, the second will be slower but generally give richer results (although it cannot do tags with too-high count)
  • the new search routine works on multiple files, so 'related tags' now shows on tag dialogs launched from a selection of thumbnails!
  • also, all the related search buttons now search any selection of tags you make!!! so if you can't remember that character's name, just click on the series or another character they are often with and hit the search, and you should get a whole bunch appear
  • I am going to keep working on this in the future. the new buttons will become the only buttons, I'll try and mitigate the prototype search limitations, add some cancel tech, move to a time-based search length like the current buttons, and I'll add more settings, including for filtering so we aren't looking up related tags for 'page:x' and so on. I'm interested in knowing how you get on with IRL data. are there too many recommendations (is the tolerance too high?)? is the sorting good (is the stuff at the top relevant or often just noise?)?

  • misc

  • all users can now copy their service keys (which are a technical non-changing hex identifier for your client's services) from the review services window--advanced mode is no longer needed. this may be useful as the client api transitions to service keys
  • when a job in the downloader search log generates new jobs (e.g. fetches the next page), the new job(s) are now inserted after the parent. previously, they were appended to the end of the list. this changes how ngugs operate, converting their searches from interleaved to sequential!
  • restarting search log jobs now also places the new job after the restarted job
  • when you create a new export folder, if you have default metadata export sidecar settings from a previous manual file export, the program now asks if you want those for the new export folder or an empty list. previously, it just assigned the saved default, which could be jarring if it was saved from ages ago
  • added a migration guide to the running from source help. also brushed up some language and fixed a bunch of borked title weights in that document
  • the max initial and periodic file limits in subscriptions is now 50k when in advanced mode. I can't promise that would be nice though!
  • the file history chart no longer says that inbox and delete time tracking are new

  • misc fixes

  • fixed a cursor type detection test that was stopping the cursor from hiding immediately when you do a media viewer drag in Qt6
  • fixed an issue where 'clear deletion record' calls were not deleting from the newer 'all my files' domain. the erroneous extra records will be searched for and scrubbed on update
  • fixed the issue where if you had the new 'unnamespaced input gives (any namespace) wildcard results' search option on, you couldn't add any novel tags in WRITE autocomplete contexts like 'manage tags'!!! it could only offer the automatically converted wildcard tags as suggested input, which of course aren't appropriate for a WRITE context. the way I ultimately fixed this was horrible; the whole thing needs more work to deal with clever logic like this better, so let me know if you get any more trouble here
  • I think I fixed an infinite hang when trying to add certain siblings in manage tag siblings. I believe this was occuring when the dialog was testing if the new pair would create a loop when the sibling structure already contains a loop. now it throws up a message and breaks the test
  • fixed an issue where certain system:filetype predicates would spawn apparent duplicates of themselves instead of removing on double-click. images+audio+video+swf+pdf was one example. it was a 'all the image types' vs 'list of (all the) image types' conversion/comparison/sorting issue

  • client api

  • **this is later than I expected, but as was planned last year, I am clearing up several obsolete parameters and data structures this week. mostly it is bad service name-identification that seemed simple or flexible to support but just added maintenance debt, induced bad implementation practises, and hindered future expansions. if you have a custom api script, please read on--and if you have not yet moved to the alternatives, do so before updating!**
  • **all `...service_name...` parameters are officially obsolete! they will still work via some legacy hacks, so old scripts shouldn't break, but they are no longer documented. please move to the `...service_key...` alternates as soon as reasonably possible (check out `/get_services` if you need to learn about service keys)**
  • **`/add_tags/get_tag_services` is removed! use `/get_services` instead!**
  • **`hide_service_names_tags`, previously made default true, is removed and its data structures `service_names_to_statuses_to_...` are also gone! move to the new `tags` structure.**
  • **`hide_service_keys_tags` is now default true. it will be removed in 4 weeks or so. same deal as with `service_names_to_statuses_to_...`--move to `tags`**
  • **`system_inbox` and `system_archive` are removed from `/get_files/search_files`! just use 'system:inbox/archive' in the tags list**
  • **the 'set_file_relationships' command from last week has been reworked to have a nicer Object parameter with a new name. please check the updated help!** normally I wouldn't change something so quick, but we are still in early prototype, so I'm ok shifting it (and the old method still works lmao, but I'll clear that code out in a few weeks, so please move over--the Object will be much nicer to expand in future, which I forgot about in v513)

  • many Client API commands now support modern file domain objects, meaning you can search a UNION of file services and 'deleted-from' file services. affected commands are

  • * /add_files/delete_files
  • * /add_files/undelete_files
  • * /add_tags/search_tags
  • * /get_files/search_files
  • * /manage_file_relationships/get_everything
  • a new `/get_service` call now lets you ask about an individual service by service name or service key, basically a parameterised /get_services
  • the `/manage_pages/get_pages` and `/manage_pages/get_page_info` calls now give the `page_state`, a new enum that says if the page is ready, initialised, searching, or search-cancelled
  • to reduce duplicate argument spam, the client api help now specifies the complicated 'these files' and now 'this file domain' arguments into sub-sections, and the commands that use them just point to the subsections. check it out--it makes sense when you look at it.
  • `/add_tags/add_tags` now raises 400 if you give an invalid content action (e.g. pending to a local tag service). previously it skipped these rows silently
  • added and updated unit tests and help for the above changes
  • client api version is now 41

  • boring optimisation

  • when you are looking at a search log or file log, if entries are added, removed, or moved around, all the log entries that have changed row # now update (previously it just sent a redraw signal for the new rows, not the second-order affected rows that were shuffled up/down. many access routines for these logs are sped up
  • file log status checking is completely rewritten. the ways it searches, caches and optimises the 'which is the next item with x status' queues is faster and requires far less maintenance. large import queues have less overhead, so the in and outs of general download work should scale up much better now
  • the main data cache that stores rendered images, image tiles, and thumbnails now maintains itself far more efficiently. there was a hellish O(n) overhead when adding or removing an item which has been reduced to constant time. this gonk was being spammed every few minutes during normal memory maintenance, when hundreds of thumbs can be purged at once. clients with tens of thousands of thumbnails in memory will maintain that list far more smoothly
  • physical file delete is now more efficient, requiring far fewer hard drive hits to delete a media file. it is also far less aggressive, with a new setting in _options->files and trash_ that sets how long to wait between individual file deletes, default 250ms. before, it was full LFG mode with minor delays every hundred/thousand jobs, and since it takes a write lock, it was lagging out thumbnail load when hitting a lot of work. the daemon here also shuts down faster if caught working during program shut down

  • boring code cleanup

  • refactored some parsing routines to be more flexible
  • added some more dictionary and enum type testing to the client api parameter parsing routines. error messages should be better!
  • improved how `/add_tags/add_tags` parsing works. ensuring both access methods check all types and report nicer errors
  • cleaned up the `/search_files/file_metadata` call's parsing, moving to the new generalised method and smoothing out some old code flow. it now checks hashes against the last search, too
  • cleaned up `/manage_pages/add_files` similarly
  • cleaned up how tag services are parsed and their errors reported in the client api
  • the client api is better about processing the file identifiers you give it in the same order you gave
  • fixed bad 'potentials_search_type'/'search_type' inconsistency in the client api help examples
  • obviously a bunch of client api unit test and help cleanup to account for the obsolete stuff and various other changes here
  • updated a bunch of the client api unit tests to handle some of the new parsing
  • fixed the remaining 'randomly fail due to complex counting logic' potential count unit tests. turns out there were like seven more of them

• version 513

  • client api

  • the Client API now supports the duplicates system! this is early stages, and what I've exposed is ugly and technical, but if you want to try out some external dupe processing, give it a go and let me know what you think! (issue #347)
  • a new 'manage file relationships' permission gives your api keys access
  • the new GET commands are:
  • - `/manage_file_relationships/get_file_relationships`, which fetches potential dupes, dupes, alternates, false positives, and dupe kings
  • - `/manage_file_relationships/get_potentials_count`, which can take two file searches, a potential dupes search type, a pixel match type, and max hamming distance, and will give the number of potential pairs in that domain
  • - `/manage_file_relationships/get_potential_pairs`, which takes the same params as count and a `max_num_pairs` and gives you a batch of pairs to process, just like the dupe filter
  • - `/manage_file_relationships/get_random_potentials`, which takes the same params as count and gives you some hashes just like the 'show some random potential pairs' button
  • the new POST commands are:
  • - `/manage_file_relationships/set_file_relationships`, which sets potential/dupe/alternate/false positive relationships between file pairs with some optional content merge and file deletes
  • - `/manage_file_relationships/set_kings`, which sets duplicate group kings
  • more commands will be written in the future for various remove/dissolve actions
  • wrote unit tests for all the commands!
  • wrote help for all the commands!
  • fixed an issue in the '/manage_pages/get_pages' call where the response data structure was saying 'focused' instead of 'selected' for 'page of pages'
  • cilent api version is now 40

  • boring misc cleanup and refactoring

  • cleaned and wrote some more parsing methods for the api to support duplicate search tech and reduce copypasted parsing code
  • renamed the client api permission labels a little, just making it all clearer and line up better. also, the 'edit client permissions' dialog now sorts the permissions
  • reordered and renamed the dev help headers in the same way
  • simple but significant rename-refactoring in file duplicates database module, tearing off the old 'Duplicates' prefixes to every method ha ha
  • updated the advanced Windows 'running from source' help to talk more about VC build tools. some old scripts don't seem to work any more in Win 11, but you also don't really need it any more (I moved to a new dev machine this week so had to set everything up again)

• version 512

  • two searches in duplicates

  • the duplicate filter page now lets you search 'one file is in this search, the other is in this search'! the only real limitation is both searches are locked to the same file domain
  • the main neat thing is you can now search 'pngs vs jpegs, and must be pixel dupes' super easy. this is the first concrete step towards my plan to introduce an optional duplicate auto resolution system (png/jpeg pixel dupes is easy--the jpeg is 99.9999% always better)
  • the database tech to get this working was actually simpler than 'one file matches the search', and in testing it works at _ok_ speed, so we'll see how this goes IRL
  • duplicate calculations should be faster in some simple cases, usually when you set a search to system:everything. this extends to the new two-search mode too (e.g. a two-search with one as system:everything is just a one-search, and the system optimises for this), however I also search complicated domains much more precisely now, which may make some duplicate search stuff work real slow. again, let me know!

  • sidecars

  • the txt importer/exporter sidecars now allow custom 'separators', so if you don't want newlines, you can use ', ' or whatever format you need

  • misc

  • when you right-click on a selection of thumbs, the 'x files' can now be 'x videos' or 'x pngs' etc.. as you see on the status bar
  • when you select or right-click on a selection of thumbs that all have duration, the status bar and menu now show the total duration of your selection. same deal on the status bar if you have no selection on a page of only durating-having media
  • thanks to the user who figured out the correct render flag, the new 'thumbnail ui-scale supersampling %' option now draws non-pixelly thumbs on 100% monitors when it is set higher (e.g. 200% thumbs drawing on 100% monitor), so users with unusual multi-monitor setups etc... should have a nicer experience. as the tooltip now says, this setting should now be set to the largest UI scale you have
  • I removed the newgrounds downloader from the defaults (this only affects new users). the downloader has been busted for a while, and last time I looked, it was not trivial to figure out, so I am removing myself from the question
  • the 'manage where tag siblings and parents apply' dialog now explicitly points users to the 'review current sync' panel

  • client api

  • a new command, /manage_pages/refresh_page, refreshes the specified page
  • the help is updated to talk about this
  • client api version is now 39

  • server management

  • in the 'modify accounts' dialog, if the null account is checked when you try to do an action, it will be unchecked. this should stop the annoying 400 Errors when you accidentally try to set it something
  • also, if you do 'add to expires', any accounts that currently do not expire will be deselected before the action too, with a brief dialog note about it

  • other duplicates improvements

  • I reworked a ton of code here, fixing a heap of logic and general 'that isn't quite what you'd expect' comparison selection issues. ideally, the system will just make more obvious human sense more often, but this tech gets a little complicated as it tries to select comparison kings from larger groups, and we might have some situations where it says '3 pairs', but when you load it in the filter it says 'no pairs found m8', so let me know how it goes!
  • first, most importantly, the 'show some random potential pairs' button is vastly improved. it is now much better about limiting the group of presented files to what you specifically have searched, and the 'pixel dupes' and 'search distance' settings are obeyed properly (previously it was fetching too many potentials, not always limiting to the search you set, and choosing candidates from larger groups too liberally)
  • while it shows smaller groups now, since they are all culled better, it _should_ select larger groups more often than before
  • when you say 'show some random potential pairs' with 'at least one file matches the search', the first file displayed, which is the 'master' that the other file(s) are paired against, now always matches the search. when you are set to the new two-search 'files match different searches', the master will always match the first search, and the others of the pairs will always match the second search. in the filter itself, some similar logic applies, so the files selected for actual comparison should match the search you inputted better.
  • setting duplicates with 'custom options' from the thumbnail menu and selecting 'this is better' now correctly sets the focused media as the best. previously it set the first file as the best
  • also, in the duplicate merge options, you can now set notes to 'move' from worse to better
  • as a side thing, the 'search distance' number control is now disabled if you select 'must be pixel dupes'. duh!

  • boring cleanup

  • refactored the duplicate comparison statement generation code from ClientMedia to ClientDuplicates
  • significantly refactored all the duplicate files calculation pipelines to deal with two file search contexts
  • cleaned up a bunch of the 'find potential duplicate pairs in this file domain' master table join code. less hardcoding, more dynamic assembly
  • refactored the duplicated 'figure out pixel dupes table join gubbins' code in the file duplicates database module into a single separate method, and rolled in the base initialisation and hamming distance part into it too, clearing out more duplicated code
  • split up the 'both files match' search code into separate methods to further clean the logic here
  • updated the main object that handles page data to the new serialisable dictionary, combining its hardcoded key/primitive/serialisable storage into one clean dict that looks after itself
  • cleaned up the type definitions of the the main database file search and fixed the erroneous empty set returns
  • I added a couple unit tests for the new .txt sidecar separator
  • fixed a bad sidecar unit test
  • 'client_running' and 'server_running' are now in the .gitignore

• version 511

  • thumbnail UI scaling

  • thumbnails can finally look good at high UI scales! a new setting in _options->thumbnails_, 'Thumbnail UI scale supersampling %', lets you tell hydrus to generate thumbnails at a particular UI scale. match it to your monitor, and your thumbnails should regenerate to look crisp
  • some users have complicated multi-monitor setups, or they change their UI scale regularly, so I'm not auto-setting this _yet_. let me know how it goes
  • sadly <100% for super-crunchy-mode doesn't work

  • unnamespaced search tags

  • _I am not really happy with this solution, since it doesn't neatly restore the old behaviour, but it does make things easier in the new system and I've fixed a related bug_
  • a new option in _services->manage tag display and search_, 'Unnamespaced input gives (any namespace) wildcard results', now lets you quickly search `*:sam*` by typing `sam`
  • fixed an issue where an autocomplete input with a total wildcard namespace, like `*:sam` was not matching to unnamespaced tags when preparing the list of tag results
  • wildcards with `*` namespace now have a special `(any namespace)` suffix, and they show with unnamespaced namespace colour

  • misc

  • fixed the client-server communication problem related to last week's SerialisableDictionary update. I messed up and forgot this object is used in network comms, which meant >=v510 clients couldn't talk to a <=509 server and _vice versa_ version swaps. now the server always kicks out an old SerialisableDictionary serialisation. I plan to remove the patch in 26 weeks, giving us more buffer time for users to update naturally
  • the recent option to turn off mouse-scroll-changes-menu-button-value is improved--now the wheel event is correctly passed up to the parent panel, so you'll scroll right through one of these buttons, not halt on it. the file sort control now also obeys this option
  • if you try to zoom a media in so that its virtual size would be >32,000px on a side, the canvas now zooms to 32k exactly. this is the max allowed zoom for technical reasons atm (I'll fix it in a future rewrite). this also fixes the 'zoom max' command, which previously would make no action if the max zoom created a virtual canvas bigger than this. also, 'zoom max' is now shown on the media viewer right-click menu
  • the 'max zoom' dimension for mpv windows and my native animation window is now 8k. seems like there are smaller technical limits for mpv, and my animation window isn't tiled, so this is to be extra safe for now
  • fixed a bug where it was possible to send the 'undelete file' signal to a file that was physically deleted (and therefore viewed in a special 'deleted files' domain). the file would obediently return to its original local file service and then throw 'missing file' warnings when the thumb tried to show. now these files are discarded from undelete consideration
  • if you are looking at physically deleted files, the thumbnail view now provides a 'clear deletion record' menu action! this is the same command as the button in _services->review services->all local files_, but just on the selection
  • fixed several taglists across the program that were displaying tags in the wrong display context and/or not sorting correctly. this mostly went wrong by setting sorted storage taglists (which normally show sibling/parent flare) as unsorted display taglists
  • file lookup script tag suggestions (as fetched from some external source) are now set to be sorted

  • file import options pre-import checking

  • _this stuff is advanced users only. normal users can rest assured that the way the client skips downloads for 'already in db/previously deleted' files now has fewer false negatives and false positives_
  • the awkwardly named advanced 'do not check url/hash to see if file already in db/previously deleted' checkboxes in file import options have been overhauled. now they are phrased in the positive ("check x to determine aid/pd?") and offer 'do not check', 'check', and the new 'check - and matches are dispositive'. the tooltip has been updated to talk about what they do. 'dispositive' basically means 'if this one hits, trust it over the other', and by default the 'hash' check remains dispositive over the URLs (this was previously hardcoded, now you can choose urls to rule in some cases).
  • there is also a new checkbox to optionally disable a component of the url checking that looks at neighbouring urls on the same file to determine url-mapping trustworthiness. this will solve or help explore some weird multi-url-mapping situations
  • also, novel SHA256 hashes no longer count as 'matches', just like a novel MD5 hash would not. this helps keep useful dispositive behaviour for known hashes but also automatically defers to urls when a site is being CDN-optimised and transfer hashes are different to api-reported ones. this fixes some watchers that have been using excess bandwidth on repeated downloads
  • fixed several problems with the url-lookup logic, particularly with the method that checks for 'file-neighbour' urls (simply, when a file-url match should be distrusted because that file has multiple urls of the same url class). it was also too aggressive on file/unknown url classes, which can legitimately have tokenised neighbours, and getting confused by http/https dupes
  • the neighbour test now remembers untrustworthy domains across different url checks for a file, which helps some subsequent direct-file-url checks where neighbours aren't a marker of file-url mapping reliability
  • the overall logic behind the hash and url lookup is cleaned up significantly
  • if you are an advanced user who has been working with me on this stuff, let me know how it goes. we erected this rats' nest through years of patches, and now I have cleaned it out. I'm confident it works better overall, but I may have missed one of your complicated situations. at the least, these new options should help us figure out quicker fixes in future

  • boring code cleanup

  • removed some old 'subject_identifier' arg parsing from various account-modification calls in the server code. as previously planned, for simplicity and security, the only identifier for these actions is now 'subject_account_key', and subject_identifier is only used for account lookups
  • improved the error handling around serialised object loading. the messages explain what happened and state object type and the versions involved
  • cleaned up some tag sort code
  • cleaned up how advanced file delete content updates work
  • fixed yet another duplicate potentials count unit test that was sometimes failing due to complex count perspective

• version 510

  • notes

  • duplicate metadata merge options now supports note merging. you can copy from worse to better or in both directions, with a couple extra conflict-resolution options that are a subset of note import options and have reasonable defaults.
  • the default note merge options are to go from worse to better for 'set as better' and both directions for 'they are the same', renaming notes on conflicts. **your existing duplicate metadata merge options will receive these settings on update, so if you don't want this, update your settings from the duplicate filter page**
  • the manage notes dialog gets copy and paste buttons. these will copy all the current notes and paste them to another instance of the panel, using the default (extend if possible, otherwise rename) conflict resolution rules
  • if an automatic system like a parser gives a note text that already exists on the file, the Note Import Options now discards it in all cases, no matter the names involved. no more automatic dupes!
  • ADVANCED: note import options (and related note add/merge operations that use it) now scan all prefix-matching note names for 'new note is already in file' and 'new note is an extension of a note already in file' tests. this improves a former fix to the 'successive parses of two sites with the same note name but different note text cause one of them to be dupe-added as (2), (3), (4), renames etc...' bug. the initial (1) rename will be scanned and recognised as 'already in file' and ignored or now extended as the settings say, just as if the desired name were hit. thanks to the reports here--I missed the logic the first time around
  • it would be nice to have 'manage notes' for multiple files at once--this is still a future goal

  • notes client api

  • the `/add_notes/set_notes` now takes some new parameters if you want to apply the adapted Note Import Options merge logic rather than figure out renames and extensions yourself
  • `/add_notes/set_notes` now returns the changes it made, which in the new mode may not be exactly what you instructed
  • added unit tests and help to reflect the above
  • client api version is now 38

  • misc

  • I fixed up how shift/ctrl/drag selection works on taglists. like with the recent thumbnail selection update, you can now 'undo' a shift-select with subsequent clicks or 'drag undo', and the list remembers what _was_ selected beforehand. ctrl-shift-select is also a more reliable 'deselect range'. both mouse drag selection and ctrl-drag selection use this logic, have fewer index bugs, and the ctrl-drag now chooses at the start whether this drag will be selection or deselection based on your initial click that started the drag. have a play with it--overall it just feels better now
  • the 'file log' menu now shows a 'reverse' command, which reverses all the imports in the log. if you want to import from oldest to newest with a typical booru, just start your downloader with file imports paused (check the cog icon), and then allow the gallery search to fully populate the list as normaly. once done, hit this new reverse and then unpause the files, and you should be good
  • any image files or thumbnails that are completely transparent and have a non-completely-black image now have their alpha channel stripped, just like files that are completely opaque. I believe the instances where this is a mistake outweigh the instances where it is legit, but let me know how we get on--maybe there are some weird mid-gif thumbs or something where this misfires. in the same thing, I reverted the 'psd thumbnails now have no transparency' change from last week. the issue where ffmpeg was sometimes being confused about psd layer masks from earlier should be fixed while letting legit transparency work correctly. the ultimate fix here will be to roll imagemagick into the program, which I am now planning and will start 'running from source' experiments with soon
  • the three 'additional fixed time...' settings in _options->downloading_ now have a max value of 3600, for extreme situation testing

  • boring code cleanup

  • updated my serialisabledict/list objects again--they can now handle bytes objects in any position. I will slowly migrate my existing hardcoded bytes serialisation and the old serialisablebytesdict to these freshly flexible classes
  • for clarity, across the code, renamed 'duplicate action options' to 'duplicate content merge options'
  • refactored duplicate content merge options initialisation, clearing the stuffed init and totuple to nicer get/set
  • broke apart how NoteImportOptions does its main note filtering for easier low-level access
  • cleaned a ton of note import options code up. the logic here was not great, now it is a bit tidier
  • undid whatever nonsense I was doing with taglist ctrl-drag-selection and cleaned up the main click and drag event handling along with its index calculation and 'what was clicked last time' record
  • fixed numerous weird logical/position index issues with the taglist and clicking/dragging

• version 509

  • misc

  • added an option 'mouse wheel can "scroll" through menu buttons' to _options->gui_. this turns off the behaviour where a mouse wheel event over, for instance, the file sort asc/desc button, will change the button's value rather than scrolling the underlying panel. if you found this annoying, you can finally turn it off!
  • fixed an annoying 'save service' bug that some users saw last week with the introduction of serverside Tag Filters. some users had an old datatype in their service data storage--a legacy issue--but the system now coerces all datatypes and direct sub-objects to a saveable format on load or update
  • the tag washing system now collapses more types of whitespace character to `space`. mostly this means tab is now converted to space, but some unicode stuff goes too
  • the hangul filler character `\u3164` is no longer permitted as a namespace or subtag. it can be in longer tags, but isn't allowed on its own (where it appears to be a blank space). (hydev saw one in the wild, probably from some cheeky post title)
  • let me know if you run across a newly invalid tag already in your system and the UI goes bananas--ideally hydrus should now catch this and either fix itself or report with a polite note, but let's see. if things go crazy, run _database->check and repair->fix invaliid tags_
  • improved some image transparency detection and slicing logic. it is more accurate and saves more memory now. also, the system that saves thumbnails will more reliably use jpegs when it doesn't need png's transparency
  • fixed some PSD thumbs showing a fully transparent transparency layer
  • fixed a bug where you could enter capital letters into the namespace colour list in 'tag presentation' options panel
  • the default twitter downloaders are all renamed to remove the confusing and technical 'syndication' label
  • 'speedcopy' is now an optional supported library. a couple users have suggested this to make network copies on Windows and Linux much faster. I'd like some advanced users who run from source to try adding it to their venvs, and we'll see how it works out IRL in different situations (you can see if it is loaded under _help->about_)
  • if you run from source, the 'advanced' setup route now offers a (t)est Qt install, which sets PySide6 6.4.1 (up from 6.3.21). feel free to try it out--it works well for me, but I want to test it more before trying to roll it to the releases
  • in a side thing, thanks to the user who walked me through setting up signed commits to github with my own PGP key. you can see my new key in the contacts help page, id 76249F053212133C, and I am now committing with it. I'm not very familiar with the sheer mechanics of this tech, so bear with me, but I'm pretty sure I can sign or encrypt something if ever needed

  • macOS build fix

  • since v505, many macOS users were unable to boot the built app. it has taken multiple rounds of back and forth with users, but we figured it out. (looks like pyoxidizer updating from 0.22.0 to 0.23.0 simply broke qtpy/Qt bindings, so we force a rollback this week)
  • also, the macOS app moves from PySide6 to PyQt6 this week. they are basically the same, but PyQt6 packages into a 258MB dmg, less than half the 548MB PySide6 one!
  • let me know if the macOS app gives any more trouble. otherwise, to the people who helped out here, thank you very much for the help!

  • mostly boring tag filter panel

  • removed the 'add' buttons; added 'delete' buttons to the simple whitelist and blacklist panels; added 'block everything' to simple blacklist panel
  • the panel now talks about the special sibling and namespace rules when you edit an explicit blacklist-mode-only filter (the tag import options blacklist works this way)
  • the 'you didn't need to add that exception' text and 'filter is too complicated for this panel' texts now show/hide rather than waste empty space
  • some of the simple-advanced interactions are better, but there's still some logical bork here. mostly stuff like when you hit the 'unnamespace' checkbox in the whitelist panel, it gets needlessly added to the 'except' column in the advanced, rather than just removed from the advanced 'exclude'. I'll fix this up in the near future
  • the two namespace checkbox lists are now sized more appropriately
  • the white/blacklist panels disable more simply and reliably

  • boring cleanup

  • the confusing 'view this file's duplicates' menu label, which was an artifact of an old submenu label, is removed. if the duplicate menu wants to present the 'view' commands for two locations, it'll title with the respective location, otherwise the commands speak for themselves, no label
  • some old 'check(er) timings' nomenclature is renamed to 'checker options' across the board
  • the hydrus serialisable dictionary now washes any nested lists or dicts to hydrus serialised equivalents, which should stop situations like the save service bug in future
  • the hydrus serialisable list can now handle a mix of hydrus serialisables and python primitives. it also washes its lists or dicts to serialisable equivalents
  • improved the data-stability of some image channel slicing
  • fixed some PIL fallback thumbnail generation, and improved its 'has transparency' png/jpeg decision-making
  • fixed the main thumbnail loader being confused at times about which thumbnail mime to load with. the check I have added is ultra-fast on data we are loading anyway, so we shouldn't notice a difference, but if you get slow thumb loads, let me know
  • fixed the media container embed buttons using the file mime rather than the thumb mime when loading thumbnails (again causing transparency issues)
  • fixed more generally bad mime handling in the thumbnail generation routine that could have caused more unusual transparency handling for clip, psd, or flash files

• version 508

  • misc

  • added a shortcut action to the 'media' set for 'file relationships: show x', where x is duplicates, potential duplicates, alternates, or false positives, just like the action buried in the thumbnail right-click menu. this actually works in both thumbs and the canvas.
  • fixed file deletes not getting processed in the duplicate filter when there were no normal duplicate actions committed in a batch. sorry for the trouble here--duplicate decisions and deletes are now counted and reported in the confirmation dialogs as separate numbers
  • as an experiment, the duplicate filter now says (+50%, -33%) percentage differences in the file size comparison statement. while the numbers here are correct, I'm not sure if this is helpful or awkward. maybe it should be phrased differently--let me know
  • url classes get two new checkboxes this week: 'do not allow any extra path components/parameters', which will stop a match if the testee URL is 'longer' than the url class's definition. this should help with some difficult 'path-nested URLs aren't matching to the right URL Class' problems
  • when you import hard drive files manually or in an import folder, files with .txt, .json, or .xml suffixes are now ignored in the file scanning phase. when hydrus eventually supports text files and arbitrary files, the solution will be nicer here, but this patch makes the new sidecar system nicer to work with in the meantime without, I hope, causing too much other fuss
  • the 'tags' button in the advanced-mode 'sort files' control now hides/shows based on the sort type. also, the asc/desc button now hides/shows when it is invalid (filetype, hash, random), rather than disable/enable. there was a bit more signals-cleanup behind the scenes here too
  • updated the 'could not set up qtpy/QtCore' error handling yet again to try to figure out this macOS App boot problem some users are getting. the error handling now says what the initial QT_API env variable was and tries to import every possible Qt and prints the whole error for each. hopefully we'll now see why PySide6 is not loading
  • cleaned up the 'old changelog' page. all the '.' separators are replaced with proper header tags and I rejiggered some of the ul and li elements to interleave better. its favicon is also fixed. btw if you want to edit 500-odd elements at a time in a 2MB document, PyCharm is mostly great. multi-hundred simultaneous edit hung for about five minutes per character, but multiline regex Find and Replace was instant
  • added a link to a user-written guide for running Hydrus on Windows in Anaconda to the 'installing' help
  • fixed some old/invalid dialog locations in the 'how to build a downloader' help

  • client api

  • a new `/get_files/file_hashes` command lets you look up any of the sha256, md5, sha1, sha512 hashes that hydrus knows about using any of the other hashes. if you have a bunch of md5 and want to figure out if you have them, or if you want to get the md5s of your files and run them against an external check, this is now possible
  • added help and unit tests for this new command
  • added a service enum to the `/get_services` Client API help
  • client api version is now 37
  • as a side thing, I rejiggered the 'what non-sha256 hash do these sha256 hashes have?' test here. it now returns a mapping, allowing for more efficient mass lookups, and it no longer creates new sha256 records for novel hashes. feel free to spam this on new sha256 hashes if you like

  • interesting serverside

  • the tag repository now manages a tag filter. admins with 'modify options' permission can alter it under the new menu command _services->administrate services->tag repo->edit tag filter_.
  • any time new tags are pended to the tag repository, they are now washed through the tag filter. any that don't pass are silently discarded
  • normal users will regularly fetch the tag filter as long as their client is relatively new. they can review it under a new read-only Tag Filter panel from _review services_. if their client is super old (or the server), account sync and the UI should fail gracefully
  • if you are in advanced mode and your client account-syncs and discovers the tag filter has changed, it will make a popup with a summary of the changes. I am not sure how spammy/annoying this will be, so let me know if you'd rather turn them off or auto-hide after two hours or something
  • future updates will have more feedback on _manage tags_ dialog and similar, just to let you know there and then if an entered tag is not wanted. also, admins who change the tag filter will be able to retroactively remove tags that apply to the filter, not just stop new ones. I'd also like some sibling hard-replace to go along with this, so we don't accidentalyl remove tags that are otherwise sibling'd to be good--we'll see
  • the hydrus server won't bug out so much at unusual errors now. previously, I ingrained that any error during any request would kick off automatic delays, but I have rejiggered it a bit so this mostly just happens during automatic work like update downloading

  • boring serverside

  • added get/set and similar to the tag repo's until-now-untouched tag filter
  • wrote a nice helper method that splays two tag filters into their added/changed/deleted rules and another that can present that in human-readable format. it prints to the server log whenever a human changes the tag filter, and will be used in future retroactive syncing
  • cleaned up how the service options are delivered to the client. previously, there would have been a version desync pain if I had ever updated the tag filter internal version. now, the service options delivered to the client are limited to python primitives, atm just update period and nullification period, and tag filter and other complex objects will have their own get calls and fail in quiet isolation
  • I fixed some borked nullification period initialisation serverside
  • whenever a tag filter describes itself, if either black or whitelist have more than 12 rules, it now summarises rather than listing every single one

• version 507

  • misc

  • fixed an issue where you could set 'all known tags' in the media-tag exporter box in the sidecars system
  • if a media-tag exporter in the sidecars system is set to an invalid (missing) tag service, the dialog now protests when you try to OK it. also, when you boot into this dialog, it will now moan about the invalid service. also, new media-tag exporters will always start with a valid local tag service.
  • Qt import error states are handled better. when the client boots, the various 'could not find Qt' errors at different qtpy and QtCore import stages are now handled separately. the Qt selected by qtpy, if any, is reported, as is the state of QT_API and whether hydrus thought it was importable. it seems like there have been a couple of users caught by something like system-wide QT_API env variables here, which this should reveal better in boot-crash logs from now on
  • all the new setup scripts in the base directory now push their location as the new CWD when they start, and they pop back to your original when they exit. you should be able to call them from anywhere now!
  • I've written a 'setup_desktop.sh' install script for Linux users to 'install' a hydrus.desktop file for the current install location to your applications directory. thanks to the user who made the original hydrus.desktop file for the help here
  • I fixed the focus when you open a 'edit predicate' panel that only has buttons, like 'has audio'/'no audio'. top button should have focus again, so you can hit enter quick
  • added updated link to hydownloader on the client api page

  • dupes apply better to groups of thumbs

  • tl;dr: when the user sets a 'copy both ways' duplicate file status on more than two thumbnails, the duplicate metadata merge options are applied better now
  • advanced explanation: previously, all merge updates were calculated before applying the updates, so when applied to a group of interconnected relationships, the nodes that were not directly connected to each other were not syncing data. now, all merge updates are calculated and applied to each pair in turn, and then the whole batch is repeated once more, ensuring two-way transitivity. for instance, if you are set to copy tags in both directions and set 'A is the best' of three files 'ABC', and B has tag 'x' and C has 'y', then previously A would get 'x' and 'y', but B would not get 'y' and C would not get 'x'. now, A gets 'x' before the AC merge is calculated, so A and C get x, and then the whole operation is repeated, so when AB is re-calculated, B now gets 'y' from the updated A. same thing if you set to archive if either file is archived--now that archived status will propagate across the whole group in one action

  • client api

  • the new 'tags' structure in `/get_files/file_metadata` now has the 'all known tags' service's tags
  • the 'file_services' structure in `/get_files/file_metadata` now states service name, type, and pretty type, like 'tags'
  • `/get_services` now says the service `type` and `type_pretty`, like 'tags'. `/get_services` may be reformatted to a service_key key'd Object at some point, since it uses an old custom human-readable service type as Object key atm and I'd rather we move to the same labels and references for everything, but we'll see
  • updated the client api help with more example result data for the above changes (and other stuff like 'all my files')
  • updated the client api unit tests to deal with the above changes
  • client api version is now 36

  • server/janitor improvements

  • I recommend server admins update their servers this week! everything old still works, but jannies who update have new abilities that won't work until you update
  • the petition processing page now has an 'account id' text field. paste an account id in there, and you'll get the petition counts just for that account! the petitions requested will also only be for that account!
  • if you get a 404 on a 'get petition' call (either due to another janitor clearing the last, or from a server count cache miscount), it no longer throws an error. instead, a popup appears for five seconds saying 'hey, there wasn't one after all, please hit refresh counts'

  • boring server improvements

  • refactored the account-fetching routine a little. some behind the scenes account identifier code, which determines an account from a mapping or file record, is now cleaner and more cleanly separated from the 'fetch account from account key' calls. account key is the master account identifier henceforth, and any content lookups will look up the account key and then do normal account lookup after. I will clean this further in the near future
  • a new server call looks up the account key from a content object explicitly; this will get more use in future
  • all the 'get number of x' server calls now support 'get number of x made by y' for account-specific counting. these numbers aren't cached, but should be fairly quick for janitorial purposes
  • same deal for petitions, the server can now fetch petitions by a particular user, if any
  • added/updated unit tests for these changes
  • general server code cleanup

• version 506

  • misc

  • the thumbnail/media viewer's right-click menu now shows all known modified dates for a file (under the top row submenu). any file downloaded in the past few months should have some extra ones, and you can see how the aggregate number is the reasonable minimum of what you have
  • added media viewer shortcut actions for 'zoom: 100/canvas fit/default'
  • like with the recent system:time update, the system:rating dialog now has nicer labels for the different numerical operators, saying 'more than' instead of '>' and so on
  • also on system:rating, the the 'rated' and 'not rated' choices are now folded into the main radio buttons. to say 'is rated in some way', select 'has rating.' to say 'not rated', set 'is' and make the rating blank. to not search that rating, select 'do not search'. I've wired up the click events here a little, too, to flip from 'do not search' to 'is' when you click and so on
  • to make it a little easier to get to, the 'view this file's relationships' submenu is bumped up a level, and the parent 'file relationships' menu is moved above the viewing stats row
  • thanks to a user, the install_dir/static dir now has an example hydrus.desktop file for Linux users. feel free to play around with it. the user taught me how this stuff works, so I'm going to try to integrate it into my setup scripts in the near future
  • I think I fixed a bug where on rare occasion the client would take 30 seconds to close while waiting on a random daemon like 'sleep check'
  • I undid last week's Windows auto-darkmode detection in a hotfix. thanks to the users who quickly notified me that this wasn't working well enough IRL. it is now opt-in, using launch parameter `--win_qt_darkmode_test`, and it applies darkmode 1 rather than 2. if there are no problems with this, then I will make 1 default and 2 opt-in, so let me know how it goes
  • the new Windows taskbar grouping identifier now only applies to the source version of the program. if you pinned the built exe to the taskbar, it was not grouping on that pin (issues #1273, #1271)
  • added a custom popup message if a subscription query comes up DEAD on the first sync. it was previously firing off the 'didn't find anything on first sync' error by accident
  • when you ok the manage options dialog, if you didn't change the thumbnail size, the thumbnail grids across the program no longer purge and regen
  • when you ok the manage options dialog, if you changed the media view options, the image tile cache now clears itself
  • when you ok the manage options dialog, if the set mpv.conf content hasn't changed, mpv is no longer told to reload it

  • sidecar paths

  • sidecars get more options regarding their file paths. it is all collected in a new 'sidecar filename' box in the normal metadata routing UI, either for sidecar importers or exporters
  • first off, a checkbox now allows you to remove the source media file's extension from the sidecar. with 'my_image.jpg', this would change the default sidecar path from 'my_image.jpg.txt' to 'my_image.txt'. I've heard the the new AI/ML artist .txt outputters use this!
  • secondly, an ADVANCED String Converter button lets you go bananas and convert the sidecar path to whatever you need using regexes or whatever
  • and lastly, it now has live test/result UI so you can put in an example media path and see what the sidecar will be. this thing is populated with sensible defaults and updates the string converter button's internal example text if you change things
  • I added some unit tests for these new features

  • client api

  • the `/get_files/file_metadata` call has several expansions:
  • a new `tags` structure shows all a file's tags in a neater, combined way. it can do everything the 'service_blah_to_blah_tags' structures do while still giving all information efficiently. please migrate to using this structure within the next eight weeks
  • `hide_service_names_tags` is now default True and deprecated. if you are still using it, please move off it; I will remove it in four weeks
  • added `hide_service_keys_tags` to do similar. it is default False for now, but I will make it True in four weeks and then delete it four weeks later just like `names`
  • the `time_modified` value is now the aggregated modified timestamp, not the local file modified timestamp
  • the new `time_modified_details` value is an Object of domain : timestamp for all known modified timestamps, by domain
  • added `thumbnail_width` and `thumbnail_height` for files that have proper thumbnails. they are a reliable prediction, but not a promise
  • added `is_deleted`, which refers to whether the file is either in the trash or has been fully deleted from the client
  • added `has_exif`, `has_human_readable_embedded_metadata` and `has_icc_profile` to the metadata Object
  • the unit tests have been updated to test these changes
  • the help has been updated to reflect these changes. also fixed up some little 'you wouldn't actually get that' issues in the mega 'file_metadata' response example
  • the client api version is now 35

  • running from source

  • if the venv activation fails in the setup script or launch script, they now stop there with an error message on all platforms
  • linux and macOS setup scripts now look to use 'python3' for initial venv setup, falling back to 'python' if that does not exist
  • updated the build scripts to always use 'python -m pip' instead of 'pip' or 'pip3' directly. this stops some weirder environments getting confused about which pip to use
  • updated the running from source help with several clarifications and little fixes and notes users have contributed

  • cleanup

  • refactored some menu templating functions from the cluttered ClientGUIMedia and ClientGUIResults to the new ClientGUIMediaMenus
  • for the new expanded modified dates stuff, cleaned up how the media 'pretty info lines' are sent to a menu
  • replaced a crash-prone emergency-error-handling dialog hook in the database migration rebalance routine with a simple popup message
  • cleaned up some bad type hints and other linter warnings
  • cleaned up some canvas zoom code
  • fixed another 'duplicates' unit test that would on rare occasion fail due to a too-specific test
  • removed a no-longer needed token declaration from the github build script that was raising a warning

• version 505

  • exif update

  • the client now has the ability to check your image files for basic human-readable metadata. sometimes this is timing data for a gif, often it is something like DPI, and for many of the recent ML-generated pngs, this is the original generating prompt. this is now viewable in the same way as EXIF, on the same panel. since this (and future expansions) are not EXIF _per se_, the overarching UI around here is broadly renamed 'embedded metadata'
  • the client now scans for and remembers if files have EXIF or human-readable embedded metadata. two predicates, 'system:image has exif' and 'system:image has human-readable embedded metadata' let you search for them. the vast majority of images have some sort of human-readable embedded metadata, so 'system:no human-readable embedded metadata' may typically be the more useful predicate in the latter case
  • the system predicate parser can handle these new system preds
  • to keep the system predicate list tidy, the new system preds are wrapped with 'has icc profile' into a meta-system predicate 'system:embedded metadata', like how 'system:dimensions' works
  • the media viewer now knows ahead of time if a media has embedded metadata. the button in the media viewer's top hover window that shows this is no longer a cog but a little text-on-window image, and it now only appears if the file has data to show. the tooltip previews whether this is EXIF, other data, or both
  • this knowledge is obviously now generated on file imports going forward, and new file maintenance jobs can retroactively scan for it
  • all your existing image files and gifs/apngs are scheduled for this work. they will catch up in the background over the coming weeks
  • the duplicate filter shows if one or both files have exif or other human-readable data. I had written off adding new 'scores' to the dupe filter panel until a full overhaul, but this was a simple copy/paste of the icc profile statement, so I snuck it in. also, these statements now only appear if for one image it is true and the other is false--no more 'they both have icc profiles m8', which is not a helpful comparison statement
  • added some unit tests for this new tech
  • a future expansion here will be to record the specific keys and values into the database so you can search specifically over those values (e.g. 'EXIF ISO level > 400', or 'has "parameters" text value')

  • misc

  • the 'reverse page drop shift behaviour' checkbox in _options->gui pages_ is replaced with four checkboxes. two govern whether page drops should chase the drop, either normally or with shift held down, and two new ones govern whether hydrus should dynamically navigate tabs as you move a media or page drag and drop over the tab bar. set them how you like!
  • a new EXPERIMENTAL checkbox just beneath these lets you change what the mouse wheel does to a row of page tabs--by default, the wheel will change tab selection, but if you often have an overloaded row (i.e. they overspill the bar width and you see the left/right arrows), you can set the wheel to _scroll/pan the bar_ instead
  • the 'if file is missing, remove record' job is now split into two--one that leaves no deletion record (old behaviour), and one that does (new). this new job lets you do some 'yes and I want it to stay gone' tasks like if you are syncing an old database backup to a newer client_files structure
  • thanks to user pointing out what was needed, turned on a beta 'darkmode detection' in Qt for Windows. if you launch the client in official Windows 'Apps darkmode' (under Windows settings->Colors), it should now start with your system darkmode colours. switching between light and dark mode while the client is running is pretty buggy (also my Explorer windows are buggy at this too jej), but this is a step forward. fingers crossed this feature matures and gets reliable multiplatform support in future (issue #756)

  • fixes

  • thanks to a user, the twitter downloader is fixed. seems like twitter (maybe due to Elon's new team?) changed one tiny name in the API we use. let's see if they change anything more significant in the coming weeks (issue #1268)
  • thanks to a user the 'gelbooru 0.1.11 file page parser' stops getting borked 'Rating: ' tags, and I fixed its source time fetch too. I'm pretty sure these broke because of the multiline string processing change a couple months ago, sorry for the trouble!
  • fixed a recent stupid typo that broke the media viewer's do an edge pan' action (issue #1266)
  • fixed an issue with the furry.booru.org url classes, which were normalising URLs to http rather than https for some accidental reason
  • I finally figured out the weird bug where the colour picker dialog would sometimes treat mouse moves as mouse drags over the colour-selection gradient box. this is due to a bug in Qt6 where if you have a stylesheet with a certain hover value set, the colour picker goes bananas. I tried many things to fix this and finally settled on a sledgehammer: if you have the offending value in your stylesheet, it now does some stuff that takes a second or two of lag to launch the colour picker and a second or two of lag to exit it. sorry, but that fixes it! if you want to skip the lag in the options dialog, set your stylesheet to 'default' for the duration (issue #1260)
  • fixed an issue where the new sidecar importer system was not correctly cleaning tags (removing extra whitespace, lowercasing) before committing them to the database! if you got hit with this, a simple restart should fix the incorrect labels (it wasn't _actually_ writing bad tags to the database), but if a restart does not fix it, please run _database->check and repair->fix invalid tags_ (issue #1264)
  • fixed an issue opening the new metadata sidecar edit UI when you had removed and replaced the original 'my tags' service
  • think I fixed a bug in the duplicate filter where if a file in the current pair is deleted (and removed from view), the index/pair tracking would desynchronise and cause an error if you attempted to rewind to the first pair
  • I fixed the reported 'committable decisions' count for duplicate filters set to do no duplicate content merge at all

  • build version woes

  • all the builds now run on python 3.9 (Linux and Windows were 3.8 previously). any users on systems too old to run 3.9 are encouraged to run from source instead
  • the linux build is rolled back to the older version of python-mpv. thanks to the users who helped me test this, and the specific user who let me know about the different version incompatibilities going on. basically we can't move to the new mpv on the Linux build for a little while, so the official release is rolling back to safe and stable. if you are on a newer Linux flavour, like 22.04, I recommend you pursue running from source, which is now easy on Linux
  • I am considering, in let's say two or three months, no longer supporting the Linux build. we'll see how well the running from source easy-setup scripts work out, but if they aren't a hassle, that really is the proper way to do things on Linux, and it'll solve many crashes and mpv issues

  • running from source is now simple and easy for everyone

  • transcribed the setup .bat files in the base directory to .sh for linux users and .command for macOS users! the 'running from source' help is updated too. all users are now welcome to try it out!
  • folded the 'setup_venv_qt5.bat' script into the main 'setup_venv.bat' script as a user choice for 'advanced' setup, and expanded it with prompts for qt5, mpv, and opencv
  • the setup files now say your python version and guide you through all choices
  • as Windows 8.1 users have reported problems with Qt6, the help and script recommendations on Qt5 are now <=8.1, not just 7. but it is easy to switch now, so if you want to play around, let me know what you discover

  • boring running from source and help gubbins

  • took the 'update' option out of the 'setup-venv.bat' script. this process was not doing what I thought it would and was not particularly useful. the script now always reinstalls after user hits Enter to continue, which is very reliable, gets newer versions of libraries when available, and almost always takes less than a minute
  • updated the github readme and website index to point obviously and directly at the getting started guide
  • took out some of the bloviating from the initial introduction page
  • updated the running from source help to talk about the new advanced setup and added a couple extra warnings
  • updated the running from source help to talk about Linux and macOS
  • if qtpy is missing at the very start of the program, a new error catch asks the user if they installed and activated their venv correctly (should also catch people who run client.py right off the bat without reading the docs)
  • deleted the old user-written help document about which packages to use with which Linux flavours, as the author says it is now out of date and modern pip as used by the scripts navigates it better nowadays
  • the setup_venv.bat now checks and informs the user if they do not have python installed
  • cleaned up the flow control of the batch files. more conditionals, fewer gotos
  • to keep the base install dir clean, moved the 'advanced' setup script's cut-up requirements.txts to a new folder under static/requirements. if you are manually setting up a venv and need unusual libraries, check them out for known good specific versions, otherwise you are set with the basic requirements.txt
  • to keep the install dir clean, moved the obscure 'build' requirements.txts to a new folder under static/requirements. these are mostly just notes for me when setting up a new test dev environment

  • cleanup and other boring stuff

  • as recommended by the pyopenssl page, I moved the server self-signed cert generation routine to 'cryptography' (which I'm pretty sure pyopenssl was just wrapping anyway). cryptography is added to the requirements.txt, but you should already have it. pyopenssl is still used by twisted, so it stays in the requirements.txts. both of these libraries remain optional and are only used by people hosting https services
  • if you load up a favourite search, the focus no longer goes to the autocomplete text box right after. hydev liked most of the focus propagation changes here but found this one incredibly annoying
  • when you are in profile mode and doing repository processing, the current speed is now printed regularly to the profile log to help see how fast the profiled jobs are at each step
  • simplified some duplicate filter code
  • the 'add tags/urls with the import' window now also shows 'cleaned' tags in the preview column for sidecar routers that go to tags
  • added some extra help text and tooltips to the new sidecar exporter UI
  • removed the weird '()' empty name component in .json exporters
  • cleaned up the namespace colour list widget in options->tag presentation. it now has proper add and delete buttons
  • refactored the colour picker button significantly and moved and merged its old wx patch code into the main object
  • the duplicate filter handles 'cannot rewind' errors better, including if the first pair is no longer viewable
  • pretty sure I fixed a long-time stupid hang in the unit tests that appeared occasionally after a 'favicon' fech test. it was due to a previous network engine shutdown test applying too broadly to test objects
  • cleaned up some edge cases in the 'which account added this file/mapping to the server?' tech, where it might have been possible, when looking up deleted content, to get another janitor account (i.e. who deleted the content), although I am pretty sure this situation was never possible to actually start in UI. if I add 'who deleted this?' tech in future, it'll be a separate specific call
  • cleaned up some specifically 'Qt6' references in the build script. the build requirements.txts and spec files are also collapsed down, with old Qt5 versions removed
  • filled out some incomplete abstract class definitions

• version 504

  • Qt5

  • as a reminder, I am no longer supporting Qt5 with the official builds. if you are on Windows 7 (and I have heard at least one version of Win 8.1), or a similarly old OS, you likely cannot run the official builds now. if this is you, please check the 'running from source' guide in the help, which will allow you to keep updating the program. this process is now easy in Windows and should be similarly easy on other platforms soon

  • misc

  • if you run from source in windows, the program _should_ now have its own taskbar group and use the correct hydrus icon. if you try and pin it to taskbar, it will revert to the 'python' icon, but you can give a shortcut to a batch file an icon and pin that to start
  • unfortunately, I have to remove the 'deviant art tag search' downloader this week. they killed the old API we were using, and what remaining open date-paginated search results the site offers is obfuscated and tokenised (no permanent links), more than I could quickly unravel. other downloader creators are welcome to give it a go. if you have a subscription for a da tag search, it will likely complain on its next run. please pause it and try to capture the best artists from that search (until DA kill their free artist api, then who knows what will happen). the oauth/phone app menace marches on
  • focus on the thumbnail panel is now preserved whenever it swaps out for another (like when you refresh the search)
  • fixed an issue where cancelling service selection on database->c&r->repopulate truncated would create an empty modal message
  • fixed a stupid typo in the recently changed server petition counting auto-fixing code

  • importer/exporter sidecar expansion

  • when you import or export files from/to disk, either manually or automatically, the option to pull or send tags to .txt files is now expanded

  • - you can now import or export URLs
  • - you can now read or write .json files
  • - you can now import from or export to multiple sidecars, and have multiple separate pipelines
  • - you can now give sidecar files suffixes, for ".tags.txt" and similar
  • - you can now filter and transform all the strings in this pipeline using the powerful String Processor just like in the parsing system
  • this affects manual imports, manual exports, import folders, and export folders. instead of smart .txt checkboxes, there's now a button leading to some nested dialogs to customise your 'routers' and, in manual imports, a new page tab in the 'add tags before import' window
  • this bones of this system was already working in the background when I introduced it earlier this year, but now all components are exposed
  • new export folders now start with the same default metadata migration as set in the last manual file export dialog
  • this system will expand in future. most important is to add a 'favourites' system so you can easily save/load your different setups. then adding more content types (e.g. ratings) and .xml. I'd also like to add purely internal file-to-itself datatype transformation (e.g. pulling url:(url) tags and converting them to actual known urls, and vice versa)

  • importer/exporter sidecar expansion (boring stuff)

  • split the importer/exporter objects into separate importers and exporters. existing router objects will update and split their internal objects safely
  • all objects in this system can now describe themselves
  • all import/export nodes now produce appropriate example texts for string processing and parsing UI test panels
  • Filename Tagging Options objects no longer track neighbouring .txt file importing, and their UI removes it too. Import Folders will suck their old data on update and convert to metadata routers
  • wrote a json sidecar importer that takes a parsing formula
  • wrote a json sidecar exporter that takes a list of dictionary names to export to. it will edit an existing file
  • wrote some ui panels to edit single file metadata migration routers
  • wrote some ui panels to edit single file metadata migration importers
  • wrote some ui panels to edit single file metadata migration exporters
  • updated edit export folder panel to use the new UI. it was already using a full static version of the system behind the scenes; now this is exposed and editable
  • updated the manual file export panel to use the new UI. it was using a half version of the system before--now the default options are updated to the new router object and you can create multiple exports
  • updated import folders to use the new UI. the filename tagging options no longer handles .txt, it is now on a separate button on the import folder
  • updated manual file imports to use the new UI. the 'add tags before import' window now has a 'sidecars' page tab, which lets you edit metadata routers. it updates a path preview list live with what it expects to parse
  • a full suite of new unit tests now checks the router, the four import nodes, and the four export nodes thoroughly
  • renamed ClientExportingMetadata to ClientMetadataMigration and moved to the metadata module. refactored the importers, exporters, and shared methods to their own files in the same module
  • created a gui.metadata module for the new router and metadata import/export widgets and panels
  • created a gui.exporting module for the existing export folder and manual export gui code
  • reworked some of the core importer/exporter objects and inheritance in clientmetadatamigration
  • updated the HDDImport object and creation pipeline to handle metadata routers (as piped from the new sidecars tab)
  • when the hdd import or import folder is set to delete original files, now all defined sidecars are deleted along with the media file
  • cleaned up a bunch of related metadata importer/exporter code
  • cleaned import folder code
  • cleaned hdd importer code

• version 503

  • misc

  • fixed show/hiding the main gui splitters after a regression in v502. also, keyboard focus after these events should now be less jank
  • thanks to a user, the Deviant Art parser we rolled back to recently now gets video support. I also added artist tag parsing like the api parser used to do
  • if you use the internal client database backup system, it now says in the menu when it was last run. this menu doesn't update often, so I put a bit of buffer in where it says 'did one recently'. let me know if the numbers here are ever confusing
  • fixed a bug where the database menu was not immediately updating the first time you set a backup location
  • if an apng has sub-millisecond frame durations (seems to be jitter-apngs that were created oddly), these are now each rounded up to 1ms. any apngs that previously appeared to have 0 duration now have borked-tiny but valid duration and will now import ok
  • the client now catches 529 error responses from servers (service is overloaded) and treats them like a 429/509 bandwidth problem, waiting for a bit before retrying. more work may be needed here
  • the new popup toaster should restore from minimised better
  • fixed a subtle bug where trashing and untrashing a file when searching the special 'all my files' domain would temporarily sort that file at the front/end of sorting by 'import time'
  • added 'dateutil present' to _help->about_ and reordered all the entries for readability
  • brushed up the network job response-bytes-size counting logic a little more
  • cleaned up the EVT_ICONIZE event processing wx/Qt patch

  • running from source is now easy on Windows

  • as I expect to drop Qt5 support in the builds next week, we need an easy way for Windows 7 and other older-OS users to run from source. I am by no means an expert at this, but I have written some easy-setup scripts that can get you running the client in Windows from nothing in a few minutes with no python experience
  • the help is updated to reflect this, with more pointers to 'running from source', and that page now has a new guide that takes you through it all in simple steps
  • there's a client-user.bat you can edit to add your own launch parameters, and a setup_help.bat to build the help too
  • all the requirements.txts across the program have had a full pass. all are now similarly formatted for easy future editing. it is now simple to select whether you want Qt5 or Qt6, and seeing the various differences between the documents is now obvious
  • the .gitignore has been updated to not stomp over your venv, mpv/ffmpeg/sqlite, or client-user.bat
  • feedback on how this works and how to make it better would be appreciated, and once we are happy with the workflow, I will invite Linux and macOS users to generate equivalent .sh and .command scripts so we are multiplatform-easy

  • build stuff

  • _this is all wizard nonsense, so you can ignore it. I am mostly just noting it here for my records. tl;dr: I fixed more boot problems, now and in the future_
  • just when I was getting on top of the latest boot problems, we had another one last week, caused by yet another external library that updated unusually, this time just a day after the normal release. it struck some users who run from source (such as AUR), and the macOS hotfix I put out on saturday. it turns out PySide6 6.4.0 is not yet supported by qtpy. since these big libraries' bleeding edge versions are common problems, I have updated all the requirements.txts across the program to set specific versions for qtpy, PySide2/PySide6, opencv-python-headless, requests, python-mpv, and setuptools (issue #1254)
  • updated all the requirements.txts with 'python-dateutil', which has spotty default support and whose absence broke some/all of the macOS and Docker deployments last week
  • added failsafe code in case python-dateutil is not available
  • pylzma is no longer in the main requirements.txt. it doesn't have a wheel (and hence needs compiler tech to pip install), and it is only useful for some weird flash files. UPDATE: with the blessed assistance of stackexchange, I rewrote the 'decompress lzma-compressed flash file' routine to re-munge the flash header into a proper lzma header and use the python default 'lzma' library, so 'pylzma' is no longer needed and removed from all requirements.txts
  • updated most of the actions in the build script to use updated node16 versions. node12 just started getting deprecation warnings. there is more work to do
  • replaced the node12 pip installer action with a manual command on the reworked requirements.txts
  • replaced most of the build script's uses of 'set-output', which just started getting deprecation warnings. there is more work to do

• version 502

  • autocomplete dropdown

  • the floating version of the autocomplete dropdown gets the same backend treatment the media hovers and the popup toaster recently received--it is no longer its own window, but now a normal widget floating inside its parent. it should look pretty much the same, but a variety of bugs are eliminated. clients with many search pages open now only have one top level window, rather than potentially hundreds of hidden ones
  • if you have turned off floating a/c windows because of graphical bugs, please try turning them back on today. the checkbox is under _options->search_.
  • as an additional consequence, I have decided to no longer allow 'floating' autocomplete windows in dialogs. I never liked how this worked or looked, overlapping the apply/cancel buttons, and it is not technically possible to make this work with the new tech, so they are always embedded in dialogs now. the related checkbox in _options->search_ is gone as a result
  • if you ok or cancel on the 'OR' buttons, focus is now preserved back to the dropdown
  • a bunch of weird interwindow-focus-juggling and 'what happens if the user's window manager allows them to close a floating a/c dropdown'-style code is cleared out. with simpler logic, some flicker jank is simply eliminated
  • if you move the window around, any displaying floating a/c dropdowns now glide along with them; previously it updated at 10fps
  • the way the client swaps a new thumbnail grid in when results are loaded or dismissed is faster and more atomic. there is less focus-cludge, and as a result the autocomplete is better at retaining focus and staying displayed as changes to the search state occur
  • the way scroll events are caught is also improved, so the floating dropdown should fix its position on scroll more smoothly and capably

  • date system predicates

  • _this affects system:import time; :modified time; and :last viewed_
  • updated the system:time UI for time delta so you are choosing 'before', 'since', and '+/- 15% of'
  • updated the system:time UI for calendar date so you are choosing 'before', 'since', 'the day of', and '+/- a month of' rather than the ugly and awkward '<' stuff
  • updated the calendar calculations with calendar time-based system predicates, so '~=' operator now does plus or minus one month to the same calendar day, no matter how many days were in that month (previously it did +/- 30 days)
  • the system predicate parser now reassigns the '=' in a given 'system:time_type = time_delta' to '~='

  • misc

  • 'sort files by import time' now sorts files correctly even when two files were imported in the same second. thanks to the user who thought of the solution here!
  • the 'recent' system predicates you see listed in the 'flesh out system pred' dialogs now have a 'X' button that lets you remove them from the recent/favourites
  • fixed the crash that I disabled some code for last week and reactivated the code. the collect-by dropdown is back to refreshing itself whenever you change the settings in _options->sort/collect_. furthermore, this guy now spams less behind the scenes, only reinitialising if there are actual changes to the sort/collect settings
  • brushed up some network content-range checking logic. this data is tracked better, and now any time a given 206 range response has insufficient data for what its header said, this is noted in the log. it doesn't raise an error, and the network job will still try to resume from the truncated point, but let's see how widespread this is. if a server delivers _more_ data than specified, this now does raise an error
  • fixed a tiny bit of logic in how the server calculates changes in sibling and parent petition counts. I am not sure if I fixed the miscount the janitors have seen
  • if a janitor asks for a petition and the current petition count for that type is miscounted, leading to a 404, the server now quickly recalculates that number for the next request
  • updated the system predicate parser to replace all underscores with whitespace, so it can accept system predicates that use_underscores_instead_of_whilespace. I don't _think_ this messes up any of the parsing except in an odd case where a file service might have an underscore'd name, but we'll cross that bridge if and when we get to it
  • added information about 'PRAGMA quick_check;' to 'help my db is broke.txt'
  • patched a unit test that would rarely fail because of random data (issue #1217)

  • client api

  • /get_files/search_files

  • fixed the recent bug where an empty tag input with 'search all' permission would raise an error. entering no search predicates now returns an empty list in all cases, no matter your permissions (issue #1250)
  • entering invalid tags now raises a 400 error
  • improved the tag permissions check. only non-wildcard tags are now tested against the filter
  • updated my unit tests to catch these cases

  • /add_tags/search_tags

  • a unit test now explicitly tests that empty autocomplete input results in no tags
  • the Client API now responds with Access-Control-Max-Age=86400 on OPTIONS checks, which should reduce some CORS pre-flight spam
  • client api version is now 34

  • misc cleanup

  • cleaned up the signalling code in the 'recent system predicate' buttons
  • shuffled some page widget and layout code to make the embedded a/c dropdown work
  • deleted a bunch of a/c event handling and forced layout and other garbage code
  • worked on some linter warnings

• version 501

  • misc

  • the Linux build gets the same 'cannot boot' setuptools version hotfix as last week's Windows build. sorry if you could not boot v500 on Linux! macOS never got the problem, I think because it uses pyoxidizer instead of pyinstaller
  • fixed the error/crash when clients running with PyQt6 (rather than the default Qt6, PySide6) tried to open file or directory selection dialogs. there was a slight method name discrepancy between the two libraries in Qt6 that we had missed, and it was sufficiently core that it was causing errors and best, crashes at worst
  • fixed a common crash caused after several options-saving events such as pausing/resuming subscriptions, repositories, import/export folders. thank you very much to the users who reported this, I was finally able to reproduce it an hour before the release was due. the collect control was causing the crash--its ability to update itself without a client restart is disabled for now
  • unfortunately, it seems Deviant Art have locked off the API we were using to get nice data, so I am reverting the DA downloader this week to the old html parser, which nonetheless still sems to work well. I expect we'll have to revisit this when we rediscover bad nsfw support or similar--let me know how things go, and you might like to hit your DA subs and 'retry ignored'
  • fixed a bad bug where manage rating dialogs that were launched on multiple files with disagreeing numerical ratings (where it shows the stars in dark grey), if okayed on that 'mixed' rating, rather than leaving them untouched, were resetting all those files back to the minimum allowed star value. I do not know when this bug came in, it is unusual, but I did do some rating state work a few weeks ago, so I am hoping it was then. I regret this and the inconvenience it has caused
  • if you manually navigate while the media viewer slideshow is running, the slideshow timer now resets (e.g. if you go 'back' on an image 7 seconds into a 10 second slideshow, it will show the previous image for 10 seconds, not 3, before moving on again)
  • fixed a type bug in PyQt hydrus when you tried to seek an mpv video when no file was loaded (usually happens when a seek event arrives late)
  • when you drop a hydrus serialised png of assorted objects onto a multi-column list, the little error where it says 'this list does not take objects of type x' now only shows once! previously, if your png was a list of objects, it could make a separate type error for each in turn. it should now all be merged properly
  • this import function also now presents a summary of how many objects were successfully imported
  • updated all ui-level ipfs multihash fetching across the program. this is now a little less laggy and uses no extra db in most cases
  • misc code and linter warning cleanup

  • tag right-click

  • the 'edit x' entry in the tag right-click menu is now moved to the 'search' submenu with the other search-changing 'exclude'/'remove' etc.. actions
  • the 'edit x' entry no longer appears when you only select invertible, non-editable predicates
  • if you right-click on a -negated tag, the 'search' menu's action label now says 'require samus aran' instead of the awkward 'exclude -samus aran'. it will also say the neutral 'invert selection' if things get complicated

  • notes logic improvements

  • if you set notes to append on conflict and the existing note already contains the new note, now no changes will be made (repeatedly parsing the same conflcting note now won't append it multiple times)
  • if you set notes to rename on conflict and the note already exists on another name, now no changes will be made (i.e. repeatedly parsing the same conflicting note won't create (1), (2), (3)... rename dupes)

  • client api

  • /add_tags/search_tags gets a new parameter, 'tag_display_type', which lets you either keep searching the raw 'storage' tags (as you see in edit contexts like the 'manage tags' dialog), or the prettier sibling-processed 'display' tags (as you see in read contexts like a normal file search page)
  • /get_files/file_metadata now returns 'ipfs_multihashes' structure, which gives ipfs service key(s) and multihashes
  • if you run /get_files/search_files with no search predicates, or with only tags that do not parse correctly so you end up with no tags, the search now returns nothing, rather than system:everything. I will likely make this call raise errors on bad tags in future
  • the client api help is updated to talk about these
  • there's also unit tests for them
  • client api version is now 33

  • popup messages

  • the background workings of the popup toaster are rewritten. it looks the same, but instead of technically being its own window, it is now embedded into the main gui as a raised widget. this should clear up a whole heap of jank this window has caused over the years. for instance, in some OSes/Window Managers, when a new subscription popup appeared, the main window would activate and steal focus. this annoying thing should, fingers crossed, no longer happen
  • I have significantly rewritten the layout routine of the popup toaster. beyond a general iteration of code cleanup, popup messages should size their width more sensibly, expand to available space, and retract better after needing to grow wide
  • unfortunately, some layout jank does remain, mostly in popup messages that change height significantly, like error tracebacks. they can sometimes take two frames to resize correctly, which can look flickery. I am still doing something 'bad' here, in Qt terms, and have to hack part of the layout update routine. let me know what else breaks for you, and I will revisit this in future
  • the 'BUGFIX: Hide the popup toaster when the main gui is minimised/loses focus' checkboxes under _options->popups_ are retired. since the toaster is now embedded into the main gui just like any search page, these issues no longer apply. I am leaving the two 'freeze the popup toaster' checkboxes in place, just so we can play around with some virtual desktop issues I know some users are having, but they may soon go too
  • the popup toaster components are updated to use Qt signals rather than borked object callables
  • as a side thing, the popup toaster can no longer grow taller than the main window size

• version 500

  • crashes

  • I messed the mpv update up in v499. my golden rule is never to put out bleeding-edge library updates, but without thinking I gave everyone a dll from late august. it turns out this thing was pretty crashy, and many users were getting other unusual behaviour as well. it seems like people on very new versions of Windows were mostly ok, but a little instability, whereas some older-Windows users were unable to start the client or could boot but couldn't load mpv at all. these latter cases were plagued with other problems. thanks to user help, we discovered it was the newer mpv dll causing all the problems, and an older one, from early May, seems to be fine
  • so, I am rolling back the mpv in the windows releases. the 'v3' 2022-08-29 I bundled in 499 was causing several users serious problems, possibly because of the advanced 'v3' chipset instructions or related advanced compiler tech. for the Qt6 release, we are going back to 2022-05-01, which several users report as stable, and for the Qt5 we are rolling back to the 498 version, 2021-02-28, which is back to mpv-1.dll. Since Qt5 users are increasingly going to be Win 7, we'll go super safe. THEREFORE, Qt5 extract users will want to perform a clean install this week: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
  • (you can alternately just delete the now-surplus mpv-2.dll in your install directory, but a full clean install is good to do from time to time, so may as well)
  • updated the sqlite dll in the windows release to 2022-05, and the exe in the db directory to 2022-09
  • rewrote how some internal MPV events are signalled to Qt. they now have their own clean custom event types rather than piggy-backing on some bad old hydrus pubsub code
  • I either fixed a rare boot crash related to the popup messaging system, maybe exclusively on macOS, or I improved it and we'll get a richer error now

  • tag sibling search

  • if you search explicitly for a tag that has a better sibling (one way this can happen is when loading up an old favourite search), the client will now auto-convert that tag to the ideal in the search code and give you results for the siblinged tag
  • this started off as a predicted five minute thing and spilled out into a multi-hour saga of me realising some tag sibling search code was A) wrong in edge cases and B) slow in edge cases. I have subtly reshaped how core file-tag search works in the client so that it consults each tag service in turn based on its siblings and its mappings, rather than mixing them together. this does not matter for 99.98% of cases, but if you have some weird overlapping siblings across different services, you should now get the correct results. also, some optimisations are more effective, so any instance of searching for tags on small tag services on 'all known tags' is now a bit quicker
  • big brain: please note the logic here is complex, and I have not yet updated autocomplete counting to handle this situation. if you type 'cat' and get 'cat (3)' from the three 'cat' tags on 'my tags', but 'cat' is siblinged to 'species:feline' on a big service like the PTR, it will still say (3), rather than (403) or whatever from the auto-corrected PTR results. I have a plan to fix this in a future cleanup round

  • tag subtags and namespace wildcards

  • searching for 'samus aran' no longer delivers files that have 'character:samus aran'. the subtag->namespace logic no longer applies. this was a fun idea from the very start of the program, but it was never all that useful as default behaviour and added several headaches, now eliminated. if you wish to perform this search going forward, please enter '*:samus aran', which is now an acceptable wildcard input
  • tag lookup is unaffected. typing 'samus aran' will still provide 'character:samus aran' as a tag to choose from
  • a heap of rinky-dink counting logic went along with this, such as providing tag search results like ('character:samus aran (100)', 'samus aran (100-105)'), where it tried to predict how many results would come with the unnamespaced search. this no longer exists, and a decent bit of CPU is now saved in any large tag search
  • wildcard searching works on similar rules now, so if you enter 'sa*s ar', you will see 'character:samus aran' as a result in the tag list, but searching for it will not give results with 'character:samus aran'. again, enter '*:sa*s ar*' to search for all namespaces (which is now provided as a quick suggestion any time you enter an unnamespaced wildcard), or enter 'character:sa*s ar*' explicitly
  • 'system:tag as number' also now follows similar rules, so if you leave the namespace field blank, it will search unnamespaced numbers. it now supports namespace wildcards, so you can enter '*' to get the old behaviour. the placeholder text on the namespace input now states this
  • 'system:number of tags' now uses the same UI as 'system:tag as number', where you enter '*' as the namespace to mean all namespaces, rather than checking a box

  • misc

  • all tag, namespace, and wildcard search predicates are now properly editable from the active search box. shift+double-click or select from the right-click menu, and you now get a simple text input alongside any system predicate panels. previously, this would only offer you a button to invert the tag to -tag and _vice versa_. now, you can add or remove the '-' and '*' characters yourself info to freely convert between tags, namespace:anything, and wildcard search predicates (issue #1235)
  • thanks to a user, you can now add '{#}' to an export filename pattern to get the '#' column in your filename (useful if you want to export files in the order they are currently in on the page)
  • furthermore, if you delete items from the manual file export window, the '#' column now recalculates itself to stay contiguous and in order (previously, it left gaps)
  • fixed a bug when deleting siblings on a local tags service. sorry for the trouble!
  • on manage siblings, when you remove, add, or replace a pair on a local tags service, you will now get a simple 'note' reason informing you more on what is going on. the 'REPLACEMENT:' thing recently added to tag repositories should now work for you too
  • when a downloader or similar adds files to a page, and you have at least one existing file selected, the status bar now updates correctly
  • fixed a critical issue that was affecting some users with damaged similar file search trees. when starting similar file search tree rebalancing maintenence, their client would go into an infinite loop and spool the cyclic branch into an ever-growing journal file in their temp directory until their system drive briefly ran out of space. sorry for the trouble, and thank you for the excellent reports that helped to figure this out (issue #1239)
  • the similar files search tree rebalance maintenance now detects more sorts of damaged trees and handles them gracefully, and the full tree regeneration clears out any damaged maintenance information too
  • fixed another problem with the tree branch maintenance system when the root was accidentally queued for branch rebalance
  • when you right-click->copy a wildcard search tag, it now copies the actual wildcard text, not the display text with (wildcard search) over the top
  • I added ',' to the list of non-decodable characters in the hacky URL Class encoding/decoding routine. sites that use an encoded comma (%_2C) for regular path components or query parameters should now work
  • a user has fixed a regex parsing problem in the predicate parser for system:hash
  • OR search predicates now sort their sub-predicates on construction/editing, meaning the label is always of set order, and they can now compare with and hence reliably nullify each other
  • the manage logins dialog now boots a little taller
  • the main gui tab bar may look a bit nicer/more appropriate in macOS
  • updated the help text on gui pages where it talks about overflowing rows of tabs, which auto-scroll even worse in Qt6, hooray

  • client api

  • the client api now handles request disconnects better. the hydrus server code benefits from the same engine improvements
  • the 'twisted.internet.defer.CancelledError' logspam is cleaned up!
  • if a client disconnects before a client api autocomplete tag search or a file search is complete, that database job is now cancelled quickly just like when you type new characters in the client UI or stop a slow search
  • if you are a client api dev, please let me know how this works out IRL. I'm not 100% sure what a 'disconnect' means in this context, but if you want to develope autocomplete quick lookup as the user types, and you have a way clientside to cancel/kill an ongoing request before it is complete, please give it a go and let me know if this all works. cancelled requests don't make a log record right now, but you should see the client's db lock free up instantly. at the very least, I have the proper infrastructure for this now, so I can add more/better 'cancel' hooks as we need them

  • uninteresting code cleanup

  • refactored the file note mapping db code to a new module
  • refactored the file service pathing db code (this does directory structures and multihashes for ipfs) to a new module
  • refactored some tag display, tag filtering, and tag autocomplete calls down to appropriate db modules
  • refactored and extended some tag sibling database methods and names to clarify whether they were working with ids or strings

• version 499

  • mpv

  • updated the mpv version for Windows. this is more complicated than it sounds and has been fraught with difficulty at times, so I do not try it often, but the situation seems to be much better now. today we are updating about twelve months. I may be imagining it, but things seem a bit smoother. a variety of weird file support should be better--an old transparent apng that I know crashed older mpv no longer causes a crash--and there's some acceleration now for very new CPU chipsets. I've also insisted on precise seeking (rather than keyframe seeking, which some users may have defaulted to). mpv-1.dll is now mpv-2.dll
  • I don't have an easy Linux testbed any more, so I would be interested in a Linux 'running from source' user trying out a similar update and letting me know how it goes. try getting the latest libmpv1 and then update python-mpv to 1.0.1 on pip. your 'mpv api version' in _help->about_ should now be 2.0. this new python-mpv seems to have several compatibility improvements, which is what has plagued us before here
  • mpv on macOS is still a frustrating question mark, but if this works on Linux, it may open another door. who knows, maybe the new version doesn't crash instantly on load

  • search change for potential duplicates

  • this is subtle and complicated, so if you are a casual user of duplicates, don't worry about it. duplicates page = better now
  • for those who are more invested in dupes, I have altered the main potential duplicate search query. when the filter prepares some potential dupes to compare, or you load up some random thumbs in the page, or simply when the duplicates processing page presents counts, this all now only tests kings. previously, it could compare any member of a duplicate group to any other, and it would nominate kings as group representatives, but this lead to some odd situations where if you said 'must be pixel dupes', you could get two low quality pixel dupes offering their better king(s) up for actual comparison, giving you a comparison that was not a pixel dupe. same for the general searching of potentials, where if you search for 'bad quality', any bad quality file you set as a dupe but didn't delete could get matched (including in 'both match' mode), and offer a 'nicer' king as tribute that didn't have the tag. now, it only searches kings. kings match searches, and it is those kings that must match pixel dupe rules. this also means that kings will always be available on the current file domain, and no fallback king-nomination-from-filtered-members routine is needed any more
  • the knock-on effect here is minimal, but in general all database work in the duplicate filter should be a little faster, and some of your numbers may be a few counts smaller, typically after discounting weird edge case split-up duplicate groups that aren't real/common enough to really worry about. if you use a waterfall of multiple local file services to process your files, you might see significantly smaller counts due to kings not always being in the same file domain as their bad members, so you may want to try 'all my files' or just see how it goes--might be far less confusing, now you are only given unambiguous kings. anyway, in general, I think no big differences here for most users except better precision in searching!
  • but let me know how you get on IRL!

  • misc

  • thank's to a user's hard work, the default twitter downloader gets some upgrades this week: you can now download from twitter lists, a twitter user's likes, and twitter collections (which are curated lists of tweets). the downloaders still get a lot of 'ignored' results for text-only tweets, but this adds some neat tools to the toolbox
  • thanks to a user, the Client API now reports brief caching information and should boost Hydrus Companion performance (issue #605)
  • the simple shortcut list in the edit shortcut action dialog now no longer shows any duplicates (such as 'close media viewer' in the dupes window)
  • added a new default reason for tag petitions, 'clearing mass-pasted junk'. 'not applicable' is now 'not applicable/incorrect'
  • in the petition processing page, the content boxes now specifically say ADD or DELETE to reinforce what you are doing and to differentiate the two boxes when you have a pixel petition
  • in the petition processing page, the content boxes now grow and shrink in height, up to a max of 20 rows, depending on how much stuff is in them. I _think_ I have pixel perfect heights here, so let me know if yours are wrong!
  • the 'service info' rows in review services are now presented in nicer order
  • updated the header/title formatting across the help documentation. when you search for a page title, it should now show up in results (e.g. you type 'running from source', you get that nicely at the top, not a confusing sub-header of that article). the section links are also all now capitalised
  • misc refactoring

  • bunch of fixes

  • fixed a weird and possible crash-inducing scrolling bug in the tag list some users had in Qt6
  • fixed a typo error in file lookup scripts from when I added multi-line support to the parsing system (issue #1221)
  • fixed some bad labels in 'speed and memory' that talked about 'MB' when the widget allowed setting different units. also, I updated the 'video buffer' option on that page to a full 'bytes value' widget too (issue #1223)
  • the 'bytes value' widget, where you can set '100 MB' and similar, now gives the 'unit' dropdown a little more minimum width. it was getting a little thin on some styles and not showing the full text in the dropdown menu (issue #1222)
  • fixed a bug in similar-shape-search-tree-rebalancing maintenance in the rare case that the queue of branches in need of regeneration become out of sync with the main tree (issue #1219)
  • fixed a bug in archive/delete filter where clicks that were making actions would start borked drag-and-drop panning states if you dragged before releasing the click. it would cause warped media movement if you then clicked on hover window greyspace
  • fixed the 'this was a cloudflare problem' scanner for the new 1.2.64 version of cloudscraper
  • updated the popupmanager's positioning update code to use a nicer event filter and gave its position calculation code a quick pass. it might fix some popup toaster position bugs, not sure
  • fixed a weird menu creation bug involving a QStandardItem appearing in the menu actions
  • fixed a similar weird QStandardItem bug in the media viewer canvas code
  • fixed an error that could appear on force-emptied pages that receive sort signals

• version 498

  • almost all the changes this week are only important to server admins and janitors. regular users can skip updating this week

  • overview

  • the server has important database and network updates this week. if your server has a lot of content, it has to count it all up, so it will take a short while to update. the petition protocol has also changed, so older clients will not be able to fetch new servers' petitions without an error. I think newer clients will be able to fetch older servers' ones, but it may be iffy
  • I considered whether I should update the network protocol version number, which would (politely) force all users to update, but as this causes inconvenience every time I do it, and I expect to do more incremental updates here in coming weeks, and since this only affects admins and janitors, I decided to not. we are going to be in awkward flux for a little bit, so please make sure you update privileged clients and servers at roughly the same time

  • server petition workflow

  • the server now maintains an ongoing fast count of its various repository metadata, such as 'number of mappings' and 'number of petitions of type x'. when you fetch petition counts, no longer will it count live and max out at 1,000, it'll give you good full numbers every time, and real fast
  • you can see the current numbers from the new 'service info' button on review services, which only appears in advanced mode. any user with an account key can see these numbers, which include number of petitions in the queue. I can make this more private if you like, but for now I think it is good if advanced users can see them all
  • in the petition processing page, sibling and parent petitions will now include both delete and add rows if the account and reason are the same. I'm aiming to get better 'full' coverage of a replace petition, so you can see and approve/deny both the add and the remove parts in one go. for fetching, these combined petitions count as 'delete' petitions, and won't appear in the 'add' petition queue
  • when users encounter an automatic conflict resolution in the manage siblings dialog, those auto-petitioned pairs are now assigned the same reason as the original conflicting pended pairs. they _should_ show up together in the new petition processing UI
  • as part of this, sibling and parent petitions are no longer filtered by namespace. you will see everything with that same account and reason in one go. let's try it out, and if it is too much, I will add filters clientside or something. since we are now starting to see add and remove together, we'll want to at least have the option to see everything

  • boring server stuff

  • the petition object is updated to handle multiple actions per petition, and the clientside petition UI is updated appropriately
  • the server tracks 'actionable' petition counts as separate to the number of raw petition rows. some of this was happening before, but the logic is improved, including clever counting of the new petitions that include both add and delete rows
  • for when my count-update logic inevitably fails, there is now a 'regen service info' entry in the 'administrate services' menu for all repositories. numbers generated will be printed to server log
  • some unusual repo upload logic is cleaned up, e.g. if a user with 'create permission' uploads a sibling or parent, any pending rows for that content will now be properly cleared)
  • fixed a stupid swap logical bug where janitors who could only moderate siblings (and not parents) were only being given parent numbers and vice versa
  • all server services now respond to /busy check. it requires no authentication and just returns 1 or 0 depending on the current lock state
  • fixed a bug where tag siblings or parents that were denied would still make a new definition record for the child/bad tag
  • with all the fine number changes, fleshed out the server unit tests with more examples of submitting and altering content and then checking for numbers afterwards. now checked are: file add, file admin delete, mapping add, mapping admin delete, mapping petition, mapping petition approve+deny, parent add, parent admin delete, parent pend, parent pend approve+deny, parent petition, parent petition approve+deny
  • significant refactoring of the tail end of server content update pipeline. more things now go through logic-harmonised update methods that ensure count is reliable
  • did some misc server db and constant enum code cleanup

  • misc

  • to match the new change in the server, in the client, tag and rating services now store their 'num_files' service info count as the new 'num_file_hashes'. existing numbers will be converted over during update
  • fixed a probably ten year old bug where 'num pending/petitioned files' had the same enum as 'num pending/petitioned mappings'. never noticed, since no service has done both those things
  • if the upload pending process fails due to an unusual permission error or similar, the pending menu should now recover and update itself (previously it stayed greyed out)

• version 497

  • misc

  • I bulked out the 'star' rating shape a bit more, since the new pentragram, while it looked better than my old 'by-eye' star, was a bit thin. if you prefer the pentagram, this is now selectable as a new shape type under manage services
  • the Windows installer is now Qt6 exclusively. there are no special update instructions, it should all just work™
  • the 'manage tag siblings/parents' dialogs now have explicit delete buttons, which should make mass-deletes a little easier to do. some of the background code is cleaned up too, and the 'add' button is moved up to the main button row
  • you can now hide all sibling and/or parent text-suffix 'decorators' in the manage tags and autocomplete dropdown taglists, with four new checkboxes under _options->tags_. the right-click menus of these lists let you temporarily show/hide too, just like 'hide/show parent rows'
  • when you change the namespace sort in the options, the existing collect-by dropdowns now update instantly (previously, existing pages needed a client restart to see any changes)
  • I updated how the media viewer 'note' hover window lays out and does its 'how tall should I be?' estimate. it fits better, being exactly just tall enough in more cases, but it still seems to have trouble with multiple notes that include wrapping text
  • added a link to the new flatpak release (easy Linux running-from-source setup) that a user made to the install help
  • fixed an issue with the new 'default' file import options when you right-click a watcher/gallery download--the 'show files' menu now correctly adapts to you having a default file import options
  • if you are set to elide page tab names, then all pages will tooltip their names on mouseover
  • new clients now start with (ctrl+page up/down) as 'move page selection left/right'

  • client api

  • the Client API routine that fetches file statuses for a given URL no longer double-checks 'already in db' results against your actual file system. this check is more appropriate to an actual working import process, so it now defaults off in the Client API
  • if you want to do this check because you are searching for missing files, you can turn it back on with the new 'doublecheck_file_system' parameter.
  • the client api help has been updated to reference this
  • the client api's Server header is now "client api/32 (497)". NOT "client api/17". it was stating the hydrus network version erroneously. it now states client api version and software version. if you are able to parse this header, it makes '/api_version' request superfluous
  • the client api version is now 32

  • multiline parsing

  • the parser now supports limited multiline parsing. the main changes are hardcoded: the formulae beneath note content parsers and those that do subsidiary page parser splitting no longer remove newlines when they parse. all the parsing UI and the test panels and so on are now aware of this and set flags in all the right places, and parsed notes are now washed through the new trimming/cleaning method, and everything _seems_ to basically work. the main remaining problems is the complicated string processing UI has mixed single/multi-line testing support. some looks great, most gets coerced to single-line just for the previewed test results
  • as an example, the default hentai foundry downloader now grabs the artist description as a multi-line note
  • the parsing sub-system that extracts cohesive strings from complex html blocks now inserts newlines at 'p' and 'br' tags

  • trying to parse clean multiline notes still caused several formatting issues this week, so I have updated the automatic note-washing routine to standardise hydrus notes in several new ways that I hope will not be too disruptive to manually written notes

  • the note washing routine now coerces all newline characters to 'backslash-n', regardless of platform
  • the note washing routine now trims each line, so no leading or trailing whitespace anywhere. I am open to changing this in future, maybe for handwritten notes where you really want an indent somewhere, but parsing from complex nested html tags is making a heap of weird extra whitespace, for which this is a clean solution
  • the note washing routine now trims newline gaps that are greater than two-newlines. you can split paragraphs by one empty line, but no more
  • there may be other issues figuring out cleanly formatted strings from nested html tags--so give it a go and let me know what you think. maybe p and br blocks should always make two newlines, so we have separated paragraphs, maybe I need to parse more blocks, like h1 and friends. any specific example html blocks would also be helpful

  • cleanup

  • refactored ClientGUIParsing to its own 'parsing' module and split everything into four less tangled files
  • cleaned up a bunch of taglist text presentation code, mostly simplicity and clarity in prep for future updates
  • updated the checker options button to use a Qt signal instead of a callable

• version 496

  • note import options

  • the client now has a system to set default note import options. it works exactly the same as default tag import options and shares the same UI, now named _network->downloaders->manage default import options_. you now set tag and/or note import options for a particular domain. I don't think you'll have to touch the note defaults until this system is really going and we learn more about what we want. I have made the initial defaults get all notes with some simple conflict resolution that won't discard any data
  • all url pages, watchers, watcher pages, gallery queries, gallery downloader pages, and subscriptions now have a note import options. by default, they are 'default'
  • the edit subscription dialog now has a button to set note import options _en masse_
  • all the behind the scenes stuff that connects and powers these systems is done. note parsing now works! advanced users, especially downloader makers, are encouraged to play around with this for real. the remaining hurdle is still multiline parsing support
  • notes now have a cleaning system before they are saved. to start with this week, they are now trimmed of leading or trailing whitespace or newlines

  • Qt6

  • the media viewer now draws correctly on UI scaled displays. If you are at >100% UI scale, it will now render images beautifully, using all available pixels, and state the correct zoom percentage. you look at a 4k image on a 4k screen, you now see 4k, no matter the UI scale. previously it was rendering at 100% UI scale coordinates and being nearest-neighbour scaled up
  • after several sad hours banging my head against font metrics, I finally discovered the magic flag needed and have improved the font quality of the thumbnail banners when you boot the client with only 100% UI scale monitors. should be anti-aliased now, although if you have a semi-transparent banner colour it may look slightly jank for reasons I still need to investigate.
  • I fixed the 'don't process the click that activates a media viewer into the shortcuts system' hook for Qt6 (and still working on Qt5). it is a little smarter now, too

  • misc

  • the new import options button is now an arrow-menu button. the secret right-click menu is no longer hidden. I also did some behind the scenes stuff to make it so all these arrow buttons spawn their menus on your cursor when you click, rather than hanging off the bottom-left corner of the button proper
  • rating stars of all shapes are now anti-aliased
  • greatly improved the shape of the 'star' rating star
  • moved the 'checker options' button on watcher highlight panels down a bit. maybe it'll get integrated into other import options one day--I am still thinking about it
  • archive/delete filters will not present 'delete from hard disk' as a final choice if the current domain is 'all local files'. I thought I fixed this a couple weeks ago, but there was a legacy issue
  • fixed some real jank logic when setting the tag domain in autocomplete dropdown widgets. this got messed up a little with recent updates to file and tag domain searching. I reworked the signal path and fixed some weird update bugs and situations where you could seemingly set 'all known files'/'all known tags'

  • boring code cleanup

  • refactored all zoom code from the media viewer canvas to the media viewer container. the canvas no longer manages zoom numbers or container size
  • refactored all container-position-tracking code from the media viewer canvas to the media viewer container and cleaned it
  • updated the media viewer container to recognise UI scaling and adapt the stated zoom to reflect the raw pixels on screen, not the device independent coordinate system
  • updated the native animation widget to recognise UI scaling, adapt its underlying renderer resolution appropriately, and draw that super-resolution frame to the canvas
  • updated the static image widget to recognise UI scaling, adapt its tile coordinate system and resolution appropriately, and scatter the ethereal powder of the cleansed ancients across the QPainter in order to stitch the arbitrarily zoomed super-resolution tiles together on a sub-pixel canvas with no visible seams
  • the animation and static image widgets also recognise changes in the current UI scale--if the current monitor changes or you move across monitors with differing UI scale
  • updated some old pubsub update calls in the canvas code to Qt signals
  • cleaned up some old const definitions in canvas code
  • refactored and simplified some test methods related to the canvas container and media show actions
  • cleaned up some old painter code and hacks to simpler alternatives
  • cleaned a tangle of file/tag domain update code in the autocomplete dropdowns
  • cleaned up some options getting/setting methods in the downloaders

• version 495

  • Qt6

  • if available, Qt6 is now the default. specifically, if the QT_API environment variable is not set, the default is now PySide6, and if that is not available, then PySide2 (Qt5). previously, the opposite was true
  • fixed a bug in last week's File Import Options default update with the new 'default' FIOs always showing 'new' files on a gallery/watcher highlight. the Presentation Import Options and the check to see if the pending local file domains actually exist now correctly look up the 'default' FIOs
  • Qt6 has much better UI scaling support than Qt5 for zooms other than 100%/200%. many Windows users are at 125%/150%, which revealed some pretty ugly thumbnails and thumb banner text in Qt6. thank you for the reports. I did my homework and read up on how this is _supposed_ to work and I have hacked pretty thumbnails at unusual UI scales. it also redraws itself correctly when I move from a 100% screen to a different one at 125%; let me know how you get on. I'm quite pleased
  • the media viewer is still slightly borked at >100%. the fix will be slightly different, but I have a plan and hope to have it sorted for next week.
  • fixed setting a mouse scroll wheel shortcut in shortcut options in Qt6
  • as a reminder, as far as I know, Windows 7 cannot run Qt6. I will be dropping the Qt5 build in a few weeks, so if you are a Windows 7 user, have a think on what you want to do--either stop updating, move hydrus to a newer OS, or run from source on Win 7/Qt5

  • note import options and note parsing

  • note parsing is ready in parts. I am rolling them out for feedback from advanced users and hope to link it all up into a working system next week!
  • the different 'x import options', previously file and tag import options, and this week adding 'note import options', are now edited through one combined button and dialog. this 'import options' button dynamically adjusts to deal with how many types of import options the importer has and will relabel and tooltip and right-click-menu itself appropriately
  • this new button and multi-edit-panel show '(is default)' status in menus and tabs for quick referral
  • if you want to play with note import options, check out the new EXPERIMENTAL menu option under _network->downloader components_. read the help and tooltips and let me know if I have missed anything simple, obvious, and important
  • I have no default system for Note Import Options set up yet, so I have not added it for real. I will do something domain-based, similar to Tag Import Options.
  • I did however write simple note parsing support. any Content Parser can now have a 'note' parsing type, with a note name. downloader creators, please feel free to play with this, although it isn't complicated and isn't plugged in yet. I think we should review what sites have parseable notes and plan for that rather than start implementing for real just yet. the main limitation is that the parsing system can't do multi-line results yet
  • I'd like to see if I can get NIO defaults going next week, and this should suddenly all lock into place. multi-line parsing may be easy or a massive pain, I'm not sure yet

  • misc

  • added two new checkboxes to _options->files and trash_ to turn off the yes/no confirmation when you copy/move file across multiple local file services
  • the 'overwrite this session?' confirmation dialog now says the session name you are overwriting
  • fixed a bug where thumbnails were not immediately updating their banner text on changes to the summary generator objects in _options->tag presentation_
  • moved the 'focus thumbnail in preview window' checkboxes from 'gui pages' options page to 'thumbnails'
  • updated the text and enabled status of the 'BUGFIX: discord DnD' stuff in _options->gui_
  • updated the job description texts in the file maintenance dialog, improving formatting and clarifying what happens in each missing/incorrect job, and what 'remove record' means precisely (it leaves no deletion record)
  • fixed a bug from last week when trying to edit your default tag import options

  • boring note import options cleanup and refactoring

  • moved ClientGUIImport code up to a new hydrus.client.gui.importing module, refactored it into multiple files, and merged in some other edit panels for various import gui
  • merged the file/tag import options buttons into one cleverer and cleaner class. changed its update callables into nicer Qt signals. wrote a new tabbed edit panel for it to work with, and replaced all old import option buttons across the program with the new system
  • fixed an issue where the 'import options' buttons (now merged) would allow you to set them as 'default' through the right-click menu even when the button was set to not allow defaults (this state occurs in the options dialog, when you _set_ what the defaults are)
  • fixed the same when you try to paste default options into the button
  • brushed up and completed the note import options object
  • wrote a 'edit note import options' panel
  • fixed a small thing where the 'string-to-string' list widget wasn't setting the custom 'value' column header name correctly

• version 494

  • QT6

  • thanks to a user's help, we are rolling out a Qt6 test build this week. we've been running Qt5 for a few years now. 6 is mostly a very large bugfix patch, and I am hopeful this update will relieve several legacy issues related to UI scale, colour support, draw flickering, and other unusual stuff. so far, it is working for me great. I'll be putting out joint 5 and 6 builds for 4-8 weeks, to iron out any big problems, and then I'll switch over to 6 releases exclusively. if you are an advanced user, please give it a go this or next week and let me know if you run into any traceback errors about deprecated method names or completely jank layout in the less used parts of the program
  • the actual changes you'll see are mostly style, just slightly different font spacing, things like that. if you have a system-baked Qt5 style that hydrus magically inherits, this will no longer work, you need to get a Qt6 version of the style (although I understand this is happening already for the popular styles, so you may already have them)
  • users on Windows 7 and similarly old OS versions are unable to run Qt6 programs, sorry!
  • I intend to keep the code 5-compatible, and users who run from source can choose whichever version of Qt they prefer, as here in the help: https://hydrusnetwork.github.io/hydrus/running_from_source.html#qt
  • the linux Qt6 build also goes up from ubuntu 18.04 to 20.04. let me know if you have any trouble, but it feels like it is time to update this too

  • file import options overhaul

  • I wanted to do note parsing this week, but when I reviewed the whole job, there wasn't enough time to do it properly. so, in prep for a cleaner introduction of 'note import options' next week, I am overhauling how the other import options do some stuff
  • all file import options now support filetype filtering! it uses the same control as system:filetype or in import folders, but with some improved logic. on update, existing import folder filetype settings will be copied down to the file import options
  • file import options now work on a similar 'default' system as tag import options. existing file import options will stay as-is, but new ones will begin in a 'use the default settings at time of import' state. those defaults are editable under _options->importing_. for now I am not adding a 'use this file import options default for this web domain' system, but it might happen in future. let's see how this all shakes out first
  • the file import options button now has a right-click menu like the tag import options button
  • the manage subscriptions panel now has a 'overwrite file import options' button to mass-set FIO
  • cleaned up a bunch of old file import and import options code

  • misc

  • system:filetype now remembers meta filetypes better. if you select 'all video', it will now still select all video even if hydev adds support for a new video type in future. also if you select 'video + animations', it'll say that rather than listing out every possible specific-type
  • fixed an issue where loading a favourite search wasn't always setting 'include current/pending' values on the buttons correct
  • fixed up a status display in the gallery downloader and watcher pages--if you pause an importer while it is doing work, it now says 'pausing...' as its status until any current jobs are finished. it was giving empty text before, as if it were finished already
  • fixed some unusual behaviour with downloader highlighting where the first query pended to an empty page was secretly highlighted for the next session load, and fixed the 'subscription gap downloader' also doing this and not obeying the normal 'highlight new downloaders if nothing already highlighted' option
  • improved the error when the 'make sure this directory exists' function runs into a file with that pathname
  • fixed a rare selection position error, maybe Qt6 only, when clicking in the thumbnail grid as it is loading

  • boring Qt6 code cleanup

  • as a side thing, I set up quick-launch environments for QtPy5, QtPy6, PySide2, and PySide6 in my IDE this week, so I can now test all these situations and jump back in time no problem in future
  • integrated a user's patch to bring us up to Qt6 compatibility and did a little more work to get it backwards compatible with older qtpy and Qt5
  • refactored the critical Qt boot setup and monkeypatching from QtPorting to a new QtInit module
  • migrated the hydrus code for keyboardModifiers, event-pos, and globalPos all to the Qt6 equivalents so the monkeypatching is always going to be on older versions looking forward
  • fiddled with QPoint and QPointF conversions a little so I _think_ Qt5 and Qt6 is always talking about the same type
  • updated build scripts and requirements.txts for the new situation
  • updated the help a bit for the new situation

• version 493

  • EXIF

  • in the first step of 'official' EXIF support, the media viewer now has a 'cog' button on the top hover, enabled when looking at a jpeg, that will check the file for EXIF data. if found, it will throw it up on a simple new window that shows EXIF id, label, and value. this is a hacked-together prototype, not super user-friendly, but it works. let me know what you think, and please send me any files that have weird EXIF that doesn't parse right but you think should. I already discovered a file with a null character that wouldn't display in UI, that sort of thing
  • GPS EXIF values are also parsed and extracted
  • made it so you can double-click a row in this new window to copy an EXIF value to clipboard
  • in the duplicate filter, if one or both files have exif data, this is now noted in the comparison statements, just like ICC profile! (issue #469)
  • obvious future extensions here will be storing 'has exif' in the database and allowing its presence to be searchable and enabling the cog button (or a nicer 'exif' button) only when there is known data to see. a subsequent step would be actually caching the data in the database for full EXIF search
  • as a side thing, we're now set up on the hydrus end to pull TIFF EXIF, but PIL doesn't seem to offer it, so we'll have to wait for a different solution there

  • fixes and misc

  • fixed a problem that made saved page file sorts reset their sort order one time on update to v492. thank you to a user for noticing this and discovering the fix, and I'm very sorry for the inconvenience of changing your session and favourite search sorts. unfortunately there is no easy fix other than rolling back to a backup and jumping forward to this version
  • fixed a v492 message display error when setting various duplicate relationships to three or more thumbnails at once. it was a stupid typo, sorry for the trouble! (issue #1199)
  • if a page tab name elides to a 'shorter...' length, it now has its full name as the tooltip
  • fixed a typo in update code error handling (issue #1192)
  • the duplicate filter page now remembers if you are 'searching immediately'/'search paused' (issue #1193)
  • if you are on non-Windows and export files manually or with an export folder to an NTFS or exFAT partition, this is now detected, and NTFS-invalid characters in the pattern-generated folders or filename are now replaced with underscores (issue #1194)
  • 'fixed' a system predicate bug in the 'OR*' advanced predicate parser--entering a logical expression that results in a negated system tag now causes an error. previously, it would strip the 'system:' and just enter the given text as an unnamespaced tag. furthermore, that dialog now reports specific error reasons when it fails to parse. I hope to improve support for negated system tags in future--some stuff, like archive/inbox, should be easy.
  • I think I fixed an instance where the archive/delete filter's confirmation dialog could present 'delete from hard disk' as an option when it wasn't appropriate
  • in an attempt to reduce the media-change flickering we've recently seen in the media viewer, I untangled a bunch of the canvas size/position code this week. I'm preparing a complete overhaul and neat Qt layout integration, which this starts. I _think_ I've made some things less flickery on occasion, but we'll see IRL. much more to do
  • added a '--profile_mode' launch argument, which allows you to capture the performance of boot and also try out profile mode on the server (although support there is very limited atm)

• version 492

  • sort and collect updates

  • for big brain users, the collect control now has a tag domain button. it only shows if you are in advanced mode (issue #572)
  • the sort control also has a tag domain button hidden behind advanced mode. it applies to system:num tags and namespace sorting
  • the collect control now appears on all import pages

  • archived file delete lock

  • the duplicate processing action code now no longer archives files that are due for deletion right before that deletion. this was hitting the archive delete lock
  • if archive delete lock is on and the 'other' file in the duplicate filter is archived, the option to 'this is better, delete the other' is now disabled
  • if you attempt to delete a delete-locked file during normal browsing, or if an automatic system like export folders wants to but fails on some, a popup is now made with a button to show the files that were filtered out so you can review the situation and fix it if you want
  • I am considering adding a dialog to say 'hey, this is locked, want to send back to inbox?' to fix these situations in a nice way, but I think this is probably a bad idea in terms of workflow, design, and my sanity given all the edge cases and potential future expansions of lock rules. maybe I'll add a simple 'delete and override lock checks' option, but a lock is a lock tbh. for now, I will focus on this better UI feedback of currently delete-locked files and make it simpler for humans to remove any locks

  • misc

  • using black magic, I have made it so the shortcuts for 'move left/right one page' 'and 'move home/end' do not dip down to the lowest level of a neighbouring page of pages for the next command. it now stays on the current tab level for three seconds after the most recent move command. this works in testing but may be jank in some IRL situations, so if this matters to you, let me know how it works out
  • fixed a bug in 'do a full metadata resync' that meant unprocessed row orphans were not being deleted, which lead to lingering 1950/2000-style processed gauges that didn't actually cause any work to be done on 'process now'
  • the duplicate filter now shows if one or both files have an icc profile. for now the score for this is always 0, neutral
  • I think I have reduced general lag on some busy clients

  • code cleaning and minor fixes

  • refactored file viewing stats management to a new database module
  • refactored file physical storage management to a new database module
  • cleaned up an ugly bridge that made inbox/archive work and moved it all to a clean new separate database module
  • improved some client file physical storage repair code, both in how it repairs and how it recovers in the current boot
  • updated the yes/no dialog texts when you apply 'not related' or 'alternates' to a selection
  • added a bunch of tooltips to the 'speed and memory' options panel. also clarified the example image sizes in number of pixels
  • improved how my grid layout propagates tooltips from the widget to the text when the widget is compound and in its own layout
  • consolidated where the delete lock test occurs to just one location for db, gui
  • added infrastructure to filter and report delete-locked files. callers no longer care about specific lock rules, opening this up to future expansion
  • cleaned and simplified some duplicate action processing code
  • cleaned up some file collect code, optimised it a bit too
  • the sort control now only changes sort type on mouse wheel events if the mouse is over that button
  • renamed 'tag search context' to 'tag context' across the program, mirroring a recent change with the location context, and gave it some bells and whistles. in future, the tag context will hold multiple tag services
  • wrote a new button to edit tag contexts

• version 491

  • system predicates

  • the advanced OR input, where you can type tags in complicated logical expressions, now supports system predicates! most system predicates are supported using their typical display strings. it uses the same engine as the client api, so check the examples here https://hydrusnetwork.github.io/hydrus/developer_api.html#get_files_search_files sorry for the delay here
  • the advanced input also runs tags better through the hydrus tag 'cleaning' process, so things like whitespace between the namespace colon and the subtag are cleaned up correctly, and invalid tags should be excluded
  • it also starts with the keyboard focus in the text input
  • and I think I fixed an issue with '!'', 'not', or '-' negation prefixes not parsing
  • highlighted the example parseable system predicate texts in the Client API help, and added 'last viewed' to it

  • misc

  • altering your services in _manage services_ no longer causes a full page refresh for all currently open search pages
  • in a related thing, if you click the file or tag domain of a file search page to be the same as it just was, you no longer get a page refresh
  • the rating widgets now show their current rating value on their tooltips
  • when setting a numerical rating by a drag, it no longer matters if your mouse strays above or below the widget--it will still set
  • the String Processing system has a new 'String Tag Filter' processing step. this applies the normal tag filtering object to your list of strings and also performs the hydrus 'tag cleaning' process on them, making them all lowercase and trimming whitespace and so on
  • the sibling/parent sync is now even more polite when told to do work in 'normal' time. this has been hitting a lot of new users really hard, so it should now really trickle work during normal time, throttling down when it hits a bump to avoid stunlocking you but also responding quickly to recent changes if you are fully synced
  • the database repair code is now better at healing damaged fast-text-search (FTS) tables. previously, in cases of partial damage to the virtual table, the repair code would error out
  • fixed a bug where certain search predicate calendar dates that are acceptable in Linux but not in Windows caused Windows to fail to load the session. if you put in 1965 as a search date, it should now revert to the current time on next load etc...
  • the test to see if a directory is writeable-to is improved and now handles Windows's Program Files directory correctly
  • improved how the boot scripts handle incorrect/bad database directory paths. the error handling works better, and it figures out a fallback location for crash.log better
  • a new button on 'review services' now lets advanced users copy the service key to the clipboard
  • the migrate tags dialog now lists file repositories, ipfs services, and 'all my files' as potential file filter domains
  • when checking it has space for a large transaction like a vacuum, hydrus now tries to check if you are running on a ramdisk or other severely space-limited temp dir and offers more text if this is true
  • updated the '4chan style thread api parser' to handle posts with multiple files, which fixes tvchan.moe and probably anything else running NPFchan
  • some logic testing around showing 'return to inbox' and the actual operation is fixed so it only applies to local files. in some weird advanced situations, you could previously send deleted files to inbox

  • new import/export framework

  • started a new modular metadata import/export pipeline. this thing starts out today by doing the work of newline-separated tags in a .txt sidecar file and will expand to do all sorts of metadata in other formats like JSON and XML. it will also, eventually, support arbitrary cross-type conversions like tags to urls or ratings to tags
  • export folders now support '.txt' sidecar tag exporting!
  • the '.txt' sidecar tag importing in import folders or manual imports is now handled by the new pipeline
  • the '.txt' sidecar exporting in the manual export dialog is now handled by the new pipeline
  • please expect the UI around '.txt' sidecar importing and exporting to change significantly in future. you'll be selecting different metadata types to import or export, make string processing steps to alter or filter what you get, and of course be able to compile it all into more complicated filetypes

  • cleanup and refactoring

  • mr bones gets two new columns to line up the numbers better
  • a bunch of export code got moved around. created a new module 'exporting', and moved ClientExporting.py to it, renaming to ClientExportingFiles.py
  • removed an old prototype for sidecar exporting and related plans for UI
  • the 'missing file folders on boot' dialog now points users to 'help my media files are broke.txt'
  • brushed up the 'help my x is broke.txt' documents in the database directory a little
  • fixed some surplus double backslashes in the help
  • a secret tiny label change/fix, let's see if anyone notices
  • cleaned up how the rating widgets manage and update rating state. it was ancient bad code
  • updated how different rating values are converted to UI text
  • misc cleanup of some free space checking code
  • fixed some bad quote characters in client api help JSON examples
  • improved some error handling for uploading pending content and sped up file uploads a little

• version 490

  • misc

  • fixed a stupid bug that meant the image caches were initialising with default values (as under _speed and memory_) until you opened and OKed the options dialog (or did some other options-refresh events). sorry for the trouble, please enjoy some smoother image browsing.
  • mr bones now shows more numbers, and in a neater table. it should be clearer what the percentages are for now, too
  • the _manage->regenerate_ thumbnail menu has additional quick maintenance commands for presence and integrity checks and regenerating data in the similar files system
  • wrote a new 'special duplicate' button for the edit shortcut set dialog. the list on this dialog doesn't allow duplicates (which meant the old 'duplicate' button was doing nothing), so this duplicates the current actions with 'incremented' shortcut keys. 'a' becomes 'b', 'ctrl+5' becomes 'ctrl+6', and so on. it doesn't always work, but if you want to make ten shortcuts for setting rating 1-10, this should help
  • fixed an issue where the thumbnail banner text and the media viewer background text was not changing size or font according to QSS stylesheet rules (issue #1173)
  • SIGTERM should now cause a clean program exit (previously it killed the GUI App but left some daemon threads alive for thirty seconds or more). unlike SIGINT, it will not ask you if you are sure you want to exit or if you would like to do shutdown maintenance--it just closes the client promptly
  • fixed a bug in last week's importer page status improvements--the hard drive import page wasn't showing all the updates it should have
  • brushed up some backup help

  • file services

  • fixed a bug where advanced users could set 'all known files'/'all known tags' on a search dropdown. this search domain is not supported
  • in the archive/delete filter, if the current location is 'all my files' and the files being deleted are only in one local file domain, the surplus 'all my files' will no longer appear at the top of the filter's commit dialog
  • the file services in the thumbnail select/remove menu are now sorted in the same order as the file domain button in search dropdowns
  • the thumbnail select/remove menus now exclude 'all my files' and 'all local files' if those choices are redundant (e.g. if you only have files in 'my files', 'all my files' will be hidden)
  • fixed some incorrect 'delete from x' actions appearing in thumbnail right-click menus

  • orphan files

  • there's a persistent processing bug some users have where some update files are missing but they won't redownload correctly. I think I fix that this week naturally so existing maintenance routines will now be able to fix it themselves after another round
  • fixed some issues related to deleting files from the repository updates file domain.
  • the 'clear orphan file records' maintenance command now fixes the 'all my files' umbrella services as well as the 'all local files' one. it also has nicer description, does some additional file-removal cleanup, and triggers a file recount if problems are found
  • moved 'clear orphan files' to the 'files' maintenance menu

• version 489

  • downloader pages

  • greatly improved the status reporting for downloader pages. the way the little text updates on your file and gallery progress are generated and presented is overhauled, and tests are unified across the different downloader pages. you now get specific texts on all possible reasons the queue cannot currently process, such as the emergency pause states under the _network_ menu or specific info like hitting the file limit, and all the code involved here is much cleaner
  • the 'working/pending' status, when you have a whole bunch of galleries or watchers wanting to run at the same time, is now calculated more reliably, and the UI will report 'waiting for a work slot' on pending jobs. no more blank pending!
  • when you pause mid-job, the 'pausing - status' text is generated is a little neater too
  • with luck, we'll also have fewer examples of 64KB of 503 error html spamming the UI
  • any critical unhandled errors during importing proper now stop that queue until a client restart and make an appropriate status text and popup (in some situations, they previously could spam every thirty seconds)
  • the simple downloader and urls downloader now support the 'delay work until later' error system. actual UI for status reporting on these downloaders remains limited, however
  • a bunch of misc downloader page cleanup

  • archive/delete

  • the final 'commit/forget/back' confirmation dialog on the archive/delete filter now lists all the possible local file domains you could delete from with separate file counts and 'commit' buttons, including 'all my files' if there are multiple, defaulting to the parent page's location at the top of the list. this let's you do a 'yes, purge all these from everywhere' delete or a 'no, just from here' delete as needed and generally makes what is going on more visible
  • fixed archive/delete commit for users with the 'archived file delete lock' turned on

  • misc

  • fixed a bug in the parsing sanity check that makes sure bad 'last modified' timestamps are not added. some ~1970-01-01 results were slipping through. on update, all modified dates within a week of this epoch will be retroactively removed
  • the 'connection' panel in the options now lets you configure how many times a network request can retry connections and requests. the logic behind these values is improved, too--network jobs now count connection and request errors separately
  • optimised the master tag update routine when you petition tags
  • the Client API help for /add_tags/add_tags now clarifies that deleting a tag that does not exist _will_ make a change--it makes a deletion record
  • thanks to a user, the 'getting started with files' help has had a pass
  • I looked into memory bloat some users are seeing after media viewer use, but I couldn't reproduce it locally. I am now making a plan to finally integrate a memory profiler and add some memory debug UI so we can better see what is going on when a couple gigs suddenly appear

  • important repository processing fixes

  • I've been trying to chase down a persistent processing bug some users got, where no matter what resyncs or checks they do, a content update seems to be cast as a definition update. fingers crossed, I have finally fixed it this week. it turns out there was a bug near my 'is this a definition or a content update?' check that is used for auto-repair maintenance here (long story short, ffmpeg was false-positive discovering mpegs in json). whatever the case, I have scheduled all users for a repository update file metadata check, so with luck anyone with a bad record will be fixed automatically in the background within a few hours of background work. anyone who encounters this problem in future should be fixed by the automatic repair too. thank you very much to the patient users who sent in reports about this and worked with me to figure this out. please try processing again, and let me know if you still have any issues
  • I also cleaned some of the maintenance code, and made it more aggressive, so 'do a full metadata resync' is now be even more uncompromising
  • also, the repository updates file service gets a bit of cleanup. it seems some ghost files have snuck in there over time, and today their records are corrected. the bug that let this happen in the first place is also fixed
  • there remains an issue where some users' clients have tried to hit the PTR with 404ing update file hashes. I am still investigating this

• version 488

  • the client now supports 'wavpack' files. these are basically a kind of compressed wav. mpv seems to play them fine too!
  • added a new file maintenance action, 'if file is missing, note it in log', which records the metadata about missing files to the database directory but makes no other action
  • the 'file is missing/incorrect' file maintenance jobs now also export the files' tags to the database directory, to further help identify them
  • simplified the logic behind the 'remove files if they are trashed' option. it should fire off more reliably now, even if you have a weird multiple-domain location for the current page, and still not fire if you are actually looking at the trash
  • if you paste an URL into the normal 'urls' downloader page, and it already has that URL and the URL has status 'failed', that existing URL will now be tried again. let's see how this works IRL, maybe it needs an option, maybe this feels natural when it comes up
  • the default bandwidth rules are boosted. the client is more efficient these days and doesn't need so many forced breaks on big import lists, and the internet has generally moved on. thanks to the users who helped talk out what the new limits should aim at. if you are an existing user, you can change your current defaults under _network->data->review bandwidth usage and edit rules_--there's even a button to revert your defaults 'back' to these new rules
  • now like all its neighbours, the cog icon on the duplicate right-side hover no longer annoyingly steals keyboard focus on a click.
  • did some code and logic cleanup around 'delete files', particularly to improve repository update deletes now we have multiple local file services, and in planning for future maintenance in this area
  • all the 'yes yes no' dialogs--the ones with multiple yes options--are moved to the newer panel system and will render their size and layout a bit more uniformly
  • may have fixed an issue with a very slow to boot client trying to politely wait on the thumbnail cache before it instantiates
  • misc UI text rewording and layout flag fixes
  • fixed some jank formatting on database migration help

• version 487

  • misc

  • updated the duplicate filter 'show next pair' logic again, mostly simplification and merging of decision making. it _should_ be even more resistant to weird problems at the end of batches, particularly if you have deleted files manually
  • a new button on the duplicate filter right hover window now appends the current pair to the parent duplicate media page (for if you want to do more processing to them later)
  • if you manually delete a file in the duplicate filter, if that file appears again in the current batch of pairs, those will be auto-skipped
  • if you manually delete a file in the duplicate filter, the actual delete is now deferred to when you commit the batch! it also undoes if you go back!
  • fixed a bug when editing the external program launch paths in the options
  • fixed an annoying delay-and-error-popup when clearing the separator field when editing a String Splitter. now the field just turns red and vetoes an OK with a nicer error text
  • also improved how string splitters report actual split errors
  • if you are in advanced mode, the _review services_ panels now have an 'id' button that lets you fetch the database service id
  • wrote a new database maintenance routine under _database->check and repair->resync tag mappings cache files_, which is a lightweight way of fixing ghost files or situations where files with a tag are neither counted nor appear in file results. this fixes these problems in a couple minutes, so for this it is much better than a full regen of the cache

  • cleanup and other boring stuff

  • the archive/delete filter now says which file domain it will be deleting from
  • if an archive/delete filter is launched on a 'multiple locations' file domain, it is now careful to only make delete records for the deleted files for the file services each one is actually in
  • renamed the 'default local file search location' option to 'fallback' and updated its tooltip a bit. this was really a hacky thing I needed to fill some gaps while rewriting from 'my files' to multiple local file services. the whole thing needs more attention to become more useful. I also fixed an issue where it could become invalid 'nothing' if you deleted a file service it was referring to (issue #1155)
  • I think I fixed a rare 'did not find info for that file' style problem when highlighting some watchers/downloaders
  • I think I have silenced some unhelpful BeautifulSoup (html parser) warnings that were spamming to the log in some situations
  • updated last week's big update to work with TRUNCATE journalling mode. I will be doing this for other big updates going forwards, since multi-GB WAL transactions cause problems for some users
  • last week's update also gives a time estimate in its pre-popup, based on 60k files per minute
  • removed some old database cache data that wasn't cleared in a previous update
  • a variety of misc UI text fixes and cleanup

• version 486

  • **This week's release is for advanced users only! I make a big change, and I want to make sure the update is fast and there are no unusual problems before rolling it out to all users.**

  • all my files

  • the client adds a new virtual file service this week, 'all my files', which is an umbrella covering all your local file domains. if you do not engage the multiple local file services system, you won't see it much, but if you do, you'll now have a convenient tool for saying 'all my stuff' without including trash and repository updates
  • **it will take a minute or two to generate this new service on update. if you have a client with millions of files, it may take a while**
  • 'all my files' now appears in the file domain selector button on your tag entry box if you have more than one local file domain. selecting this searches the union of all your local file domains with fast and precise count (as opposed to 'multiple locations' of the full union, which will have imprecise counts and be slower). it also does duplicate file work laser-fast (again, unlike 'multiple locations', which is often slow due to UNION complexity)
  • 'all my files' also appears in review and manage services, very similarly to 'all local files'
  • a heap of hacks I instituted when getting multiple local file services ready are now replaced with this clean 'yeah this file is valued and worth looking at' domain. for instance, downloader pages now view files in this way.
  • mr bones and the file history chart also use 'all my files', and are significantly faster to calculate. the chart also excludes repo update files and trash now
  • calls to delete or undelete on 'all my files' (this is mostly Client API and some 'default' situations) will be converted to a blanket 'force send to trash' and 'force undelete all deleted records'
  • the 'undelete files?' dialog is now a button selection dialog. it also now has an 'all the above' option when more than one local service may apply, which tells the client to undelete to all services the files have been deleted from
  • updated multiple local file services help to talk a little about the new domain
  • rearranged the sort in a couple of places where the different local file services appear. they should now be: local file domains, all my files, trash, repo updates, all local files
  • ADVANCED: the 'presentation import options' under 'file import options' now allows a full-fledged location context using the new multiple local file services system rather than the previous 'in your files(and trash too)' choice. it defaults to the new 'all my files' domain

  • misc

  • thanks to a user, the 'getting started with downloading' help has had a full pass. if you have had trouble with downloaders, particularly if you are unsure about what file import options are for, or what subscriptions are, please check it out!
  • the 'media viewers' shortcut set gets three new zoom actions: 'switch between 100% and max', 'switch between canvas and max', and 'zoom to max' (issue #1141)
  • if a media type is set to do 'exact zooms', it will now not exceed the otherwise specified max zoom
  • the file sort widget will now preserve ascending/descending status on sort type changes (rather than resetting to default) if the asc/desc strings do not change. so, if you are on 'import time'/'oldest first', and switch to 'archive time', it will now stay on 'oldest' rather than resetting to 'newest'
  • the manage tag siblings dialog now tries to automatically break loops for you, just like it will automatically break A->B, A->C conflicts. this works on manual entry or mass import
  • the manage tag siblings dialog now shows the stated 'reason' for any pair change (e.g. "AUTO-PETITION TO BREAK LOOP") in the 'note' column
  • the 'short' animation scanbar--when your mouse is away--now keeps a short disabled volume button beside it. I found it very annoying how the scan nub would jump a few pixels left/right as this popped up and down, so now it is the same width big and small
  • right-clicking on files when in pages with 'multiple locations' file domains is now much much faster
  • the filename tagging dialog now starts with the 'tags for all' focused, and the 'press up/down on empty input' shortcuts are now plugged in, so pressing up/down will change service
  • I believe I may have completely eliminated the additional superlag that sometimes occurs when adding or deleting a service. it was a database maintenance routine getting carried away with other outstanding work
  • move/add actions in the new multiple local file system now operate asynchronously and politely, spreading their work time out when the client is busy, and for large jobs they will also make a cancellable progress popup
  • cleaned up how the autocomplete entry sends some of its signals to other parts of the program
  • did some misc help and code edits/refactoring, including brushing up the Windows install section with more advanced options
  • removed the 'hydrus zooms big bad' warning from the 'media' options page. hydrus zooms big good now!

  • some database stuff

  • tl;dr: database cleans up after itself better now
  • some users have had trouble with database journal files (the 'wal' files in your db directory) on certain clients getting huge after lots of work, multiple GB, and causing the OS a headache if the journal is doing work through a computer sleep. these journals are 'supposed' to checkpoint and clean themselves up naturally, but I think a busy database chokes them. therefore, I have improved the hydrus maintenance this week: 1) the 'journal size limit' PRAGMA, which applies softly after every 30 seconds or so, is now 128MB down from 1GB. 2) databases in PERSIST (rare) mode will now specifically zero out their journal fifteen minutes. 3) databases in WAL mode (the default), in addition to regular PASSIVE checkpointing now every five minutes, will force an additional TRUNCATE checkpoint every fifteen. this should force a regular full flush and maybe help some other problems like gigantic memory bloat the same users sometimes saw. if you are a very advanced user and do active debug on the database while hydrus is using it, please note this new TRUNCATE command is aggressive and may block itself or you inconveniently. let me know how you get on!
  • moved the recent 'be careful of usb drives' section in 'installing' help to 'help my db is broke.txt'. it is very likely this problem was related to the above WAL stuff, and it was not just usb drives, I rewrote it as generalised help for anyone who gets 'delayed write failed' errors at the OS level
  • massively optimised several critical duplicate files filtering methods if the current location context has more than one file domain, and I think I cleared out the basic 'get duplicate info for this file' call of all slow calls in complex location contexts
  • the repair routine that regenerates mapping caches if any tables are missing on boot is now more reliable and covers the entirety of the mappings cache system using the new modules system. it also now regenerates just for the tag services with missing tables, not the whole cache
  • if multiple types of mapping cache tables are missing on boot, and multiple waves of regenerations covering different areas are planned, duplicate regenerations will now be skipped

• version 485

  • multiple local file services

  • multiple local file services are now available for everyone! you no longer need to be in advanced mode to create them. all are welcome, but in terms of skill level, I most recommend it for users who are comfortable with tag siblings and parents
  • the tl;dr: you can now have more than one 'my files', which lets you put things in isolated locations
  • I wrote a proper help document on multiple local file services--what they are, how they work, my recommendations, and a bit of extra info about hydrus file search in general, right here: https://hydrusnetwork.github.io/hydrus/advanced_multiple_local_file_services.html
  • file searches in 'multiple locations' on large clients are now massively faster in almost all situations. the only place multiple location searches are still slow is whenever the duplicates system (system:file relationships) comes into play

  • misc

  • in the page tab menu, you can now sort pages by total file size
  • the 'force system:limit for all searches' option is moved from the 'speed and memory' to 'search' panel
  • when files download from sites, if the raw file is served by cloudflare and has a timestamp radically different to a parsed source time, that CF timestamp is saved under a different domain rather than overwriting the original domain timestamp. this seemed to affect danbooru on about 1 in 10-20 files. note this does not change much at the moment, but when you can see and sort on individual domain modified dates, this should improve the sort
  • updated the 'installing' help to talk about bad install locations for the database. network locations are bad, and thanks to user reports, we now know USB drives can be bad if the database is busy when the OS goes to sleep
  • if a 'database is malformed' error occurs on boot, the client now recognises it and points the user to 'install_dir/db/help my db is broke.txt' for the next steps

  • boring code cleanup

  • another 60KB or so of code pulled out of ClientDB.py

  • created a new database module for url mappings and refactored various fetch and update routines to it
  • created a new database module for some rich file metadata and refactored some file filtering, history, and status testing code to it
  • created a new database module for file searching and moved all tag-based file searching code to it
  • moved several other misc methods down to database modules

• version 484

  • misc

  • fixed the simple delete files dialog for trashed files. due to a logical oversight, the simple version was not testing 'trashed' status and so didn't see anything to permanently delete and would immediately dump out. now it shows the option for trashed files again, and if the selection includes trash and non-trash, it shows multiple options
  • fixed an error in the 'show next pair' logic of the new duplicate filter queue where if it needed to auto-skip through the end of the current batch and load up the next batch (issues #1139, #1143)
  • a new setting on _options->media_ now lets you set the scanbar to be small and simple instead of hidden when the mouse is moved away. I liked this so much personally it is now the default for new users. try it out!
  • the media viewer's taglist hover window will now never send a mouse wheel event up to the media viewer canvas (so scrolling the tags won't accidentally do previous/next if you hit the end of the list scrollbar)
  • I think I have fixed the bug where going on the media viewer from borderless fullscreen to a regular window would not trigger a media container resize if the media perfectly fitted the ratio of the fullscreen monitor!
  • the system tray icon now has minimise/restore entries
  • to reduce confusion, when a content parser vetoes, it now prepends the file import 'note' with 'veto: '
  • the 'clear service info cache' job under _database->regenerate_ is renamed to 'service info numbers' and now has a service selector so you can, let's say, regen your miscounted 'number of files in trash' count without triggering a complete recount of every single mapping on the PTR the next time you open review services
  • hydrus now recognises most (and maybe all) windows executables so it can discard them from imports confidently. a user discovered an interesting exe with embedded audio that ffmpeg was seeing as an mp3--this no longer occurs
  • the 'edit string conversion step' dialog now saves a new default (which is used on 'add' events) every time you ok it. 'append extra text' is no longer the universal default!
  • the 'edit tag rule' dialog in the parsing system now starts with the tag name field focused
  • updated 'getting started/installing' help to talk more about mpv on Linux. the 'libgmodule' problem _seems_ to have a solid fix now, which is properly written out there. thanks to the users who figured all this out and provided feedback

  • multiple local file services

  • the media viewer menu now offers add/move actions just like the thumb grid
  • added a new shortcut action that lets you specify add/move jobs. it is available in the media shortcut set and will work in the thumbnail grid and the media viewer
  • add/move is now nicer in edge cases. files are filtered better to ensure only local media files end up in a job (e.g. if you were to try to move files out of the repository update domain using a shortcut), and 'add' commands from trashed files are naturally and silently converted to a pure undelete

  • boring code cleanup

  • refactored the UI side of multiple local file services add/move commands. various functions to select, filter, and question the user on actions are now pulled to a separate simple module where other parts of the UI can also access them, and there is now just one isolated pipeline for file service add/move content updates.
  • if a 'move' job is started without a source service and multiple services could apply, the main routine will now ask the user which to use using a selector that shows how many files each choice will affect
  • also rewrote the add/move menu population code, fixed a couple little issues, and refactored it to a module the media viewer canvas can use
  • wrote a new menu builder that can place a list of items either as a single item (if the list is length 1), or make a submenu if there are more. it drives the new add/move commands and now the behind the scenes of all other service-based menu population

• version 483

  • multiple local file services

  • the multiple local file services feature is ready for advanced users to test out! it lets you have more than one 'my files' service to store things, which will give us some neat privacy and management tools in future. there is no nice help for this feature yet, and the UI is still a little non-user-friendly, so please do not try it out unless you have been following it. and, while this has worked great in all testing, I generally do not recommend it for heavy use on a real client either, just in case something does go wrong. with those caveats, hit up _manage services_ in advanced mode, and you can now add new 'local file domain' services. it is possible to search, import to, and migrate files between these and everything basically works. I need to do more UI work to make it clear what is going on (for instance, I think we'll figure out custom icons or similar to show where files are), and some more search tech, and write up proper help, and figure out easy client merging so users can combine legacy clients, but please feel free to experiment wildly on a fresh client or carefully on your existing one
  • if you have more than one local file service, a new 'files' or 'local services' menu on thumbnail right-click handles duplicating and moving across local services. these actions will preserve original import times (e.g. if you move from A to B and then back to A), so they should be generally non-destructive, but we may want to add some advanced tools in future. let me know how this part goes--I think we'll probably want a different status than 'deleted from A' when you just move A->B, so as not to interfere with some advanced queries, but only IRL testing will show it
  • if you have a 'file import options' that imports files to multiple local services but the file import is 'already in db', the file import job will now examine if and where the file is still needed and send content update calls to fill in the gaps
  • the advanced delete files dialog now gives a new 'delete from all and send to trash' option if the file is in multiple local file domains
  • the advanced delete files dialog now fully supports file repositories
  • cleaned up some logic on the 'remember action' option of the advanced file deletion dialog. it also supports remembering specific file domains, not just the clever commands like 'delete and leave no record'. also this dialog no longer places the 'suggested' file service at the top of the radio button list--instead it selects that 'suggested' if there is no 'remember action' initial selection applicable. the suggested file service is now also set by the underlying thumbnail grid or media canvas if it has a simple one-service location context
  • the normal 'non-advanced' delete files dialog now supports files that are in multiple local file services. it will show a part of the advanced dialog to let you choose where to delete from

  • misc

  • thanks to user submissions, there is a bit more help docs work--for file search, and for some neat new 'mermaid' svg diagrams in siblings/parents, which are automatically generated from a markup and easy to edit
  • with the new easy-to-edit mermaid diagrams, I updated the unhelpful and honestly cringe examples in the siblings and parents help to reflect real world PTR data and brushed up all the text in the top sections
  • just a small thing--the 'pages' menu and the page picker dialog now both say 'file search' to refer to a page that searches files. previously, 'search' or 'files' was used in different places
  • completely rewrote the queue code behind the duplicate filter. an ancient bad idea is now replaced with something that will be easier to work with in future
  • you can now go 'back' in the duplicate filter even when you have only done skips so far
  • the 'index string' of duplicate filters, where it says 53/100, now also says the number of decisions made
  • fixed some small edge case bugs in duplicate filter forward/backward move logic, and fixed the recent problem with going back after certain decisions
  • updated the default nijie.info parser to grab video (issue #1113)
  • added in a user fix to the deviant art parser
  • added user-made Mega URL Classes. hydrus won't support Mega for a long while, but it can recognise and categorise these URLs now, presenting them in the media viewer if you want to open them externally
  • fixed Exif image rotation for images that also have ICC Profiles. thanks to the user who provided great test images here (issue #1124)
  • hitting F5 or otherwise saying 'refresh' explicitly will now turn a search page that is currently in 'searching paused' to 'searching immediately'. previously it silently did nothing
  • the 'current file info' in the media window's top hover and the status bar of the main window now ignores Deletion reason, and also file modified date if it is not substantially different from another timestamp already stated. this data can still be seen on the file's right-click menu's expanded info lines off the top entry. also, as a small cleanup, it now says 'modified' and 'archived' instead of 'file modified/archived', just to save some more space
  • like the above 'show if interesting' check for modified date, that list of file info texts now includes the actual import time if it is different than other timestamps. (for instance, if you migrate it from one service to another some time after import)
  • fixed a sort error notification in the edit parser dialog when you have two duplicate subsidiary parsers that both have vetoes
  • fixed the new media viewer note display for PyQt5
  • fixed a rare frame-duration-lookup problem when loading certain gifs into the media viewer

  • boring code cleanup

  • cleaned up search signalling UI code, a couple of minor bugs with 'searching immediately' sometimes not saving right should be fixed
  • the 'repository updates' domain now has a different service type. it is now a 'local update file domain' rather than a 'local file domain', which is just an enum change but marks it as different to the regular media domains. some code is cleaned up as a result
  • renamed the terms in some old media filtering code to make it more compatible with multiple local file services
  • brushed up some delete code to handle multiple local file services better
  • cleaned up more behind the scenes of the delete files dialog
  • refactored ClientGUIApplicationCommand to the widgets module
  • wrote a new ApplicationCommandProcessor Mixin class for all UI elements that process commands. it is now used across the program and will grow in responsibility in future to unify some things here
  • the media viewer hover windows now send their application commands through Qt signals rather than the old pubsub system
  • in a bunch of places across the program, renamed 'remote' to 'not local' in file status contexts--this tends to make more sense to people at out the gate
  • misc little syntax cleanup

• version 482

  • misc

  • fixed the stupid taglist scrolled-click position problem--sorry! I have a new specific weekly test for this, so it shouldn't happen again (issue #1120)
  • I made it so middle-clicking on a tag list does a select event again
  • the duplicate action options now let you say to archive both files regardless of their current archive status (issue #472)
  • the duplicate filter is now hooked into the media prefetch system. as soon as 'A' is displayed, the 'B' file will now be queued to be loaded, so with luck you will see very little flicker on the first transition from A->B.
  • I updated the duplicate filter's queue to store more information and added the next pair to the new prefetch queue, so when you action a pair, the A of the next pair should also load up quickly
  • boosted the default sizes of the thumbnail and image caches up to 32MB and 384MB (from 25/150) and gave them nicer 'bytes quantity' widgets in the options panel
  • when popup windows show network jobs, they now have delayed hide. with luck, this will make subscriptions more stable in height, less flickering as jobs are loaded and unloaded
  • reduced the extremes of the new auto-throttled pending upload. it will now change speed slower, on less strict of a schedule, and won't go as fast or slow max
  • the text colour of hyperlinks across the program, most significantly in the top-right media hover window, can now be customised in QSS. I have set some ok defaults for all the QSS styles that come with the client, if you have a custom QSS, check out my default to see what you need to do. also hyperlinks are no longer underlined and you can't 'select' their text with the mouse any more (this was a weird rich-text flag)
  • the client api and local booru now have a checkbox in their manage services panel for 'normie-friendly welcome page', which switches the default ascii art for an alternate
  • fixed an issue with the hydrus server not explicitly saying it is utf-8 when rendering html
  • may have fixed some issues with autocomplete dropdowns getting hung up in the wrong position and not fixing themselves until parent resize event or similar

  • code cleanup

  • about 80KB of code moved out of the main ClientDB.py file

  • refactored all combined files display mappings cache code from the code database to a new database module
  • refactored all combined files storage mappings cache code from the code database to a new database module
  • refactored all specific storage mappings cache code from the code database to a new database module
  • more misc refactoring of tag count estimate, tag search, and other code down to modules
  • hooked up specific display mappings cache to the repair system correctly--it had been left unregistered by accident
  • some misc duplicate action options code cleanup
  • migrated some ancient pause states--repository, subscriptions, import&export folders--to the newer options structure
  • migrated the image and thumbnail cache sizes to the newer options structure
  • removed some ancient db and dialog code from the retired dumper system

• version 481

  • fixes and improvements after last week's hover and note work

  • fixed the text colour behind the top middle hover window
  • stopped clicks on the taglist and hover greyspace being duplicated up to the main canvas (this affected the archive/delete and duplicate filter shortcuts)
  • fixed the background colour of the hover windows when using non-default stylesheets
  • fixed the notes hover window--after having shown some notes--could then lurk in the top-left corner when it should have been hidden completely
  • cleaned up some old focus test logic that weas used when hovers were separate windows
  • rewrote how each note panel in the new hover is stored. a bunch of sizing and event handling code is less hacked
  • significantly improved the accuracy of the 'how high should the note window be?' calculation, so notes shouldn't spill over so much or have a bunch of greyspace below
  • right- or middle-clicking a note now hides its text. repeat on its name to restore. this should persist through an edit, although it won't be reflected in the background atm. let's see how it works as a simple way to quickly browse a whole stack of big notes
  • a new 'notes' option panel lets you choose if you want the text caret to start at the beginning or end of the document when editing
  • you can now double-click a note tab in 'edit notes' to rename the note. some styles may let you double-click in note greyspace to create a new note, but not all will handle this (yet)
  • as an experiment, all the buttons on the media viewer hover windows now do not take focus when you click them. this should let you, for instance, click a duplicate filter processing button and then use the arrow keys and space to continue to navigate. previously, clicking a button would focus it, and navigation keys would be intercepted to navigate the 'form' of the buttons on the hover window. you can still focus buttons with tab. if this affects you, let me know how this goes!

  • misc

  • added checkboxes to _options->gui pages_ to control whether ctrl- and shift- selects will highlight media in the preview viewer. you can choose to only do it for files with no duration if you prefer
  • the 'advanced mode' tag autocomplete dropdown now has 'OR' and 'OR*' buttons. the former opens a new empty OR search predicate in the edit dialog, the latter opens the advanced text parser as before
  • the edit OR predicate panel now starts wider and with the text box having focus
  • hydrus is now more careful about deciding whether to make a png or a jpeg thumbnail. now, only thumbnails that have an alpha channel with interesting data in it are saved to png. everything else is jpeg
  • when uploading to a repository, the client will now slow down or speed up depending on how fast things are going. previously it would work on 100 mappings at a time with a forced 0.1s wait, now it can vary between 1-1,000 weight
  • just to be clean, the current files line on the file history chart now initialises at 0 on your first file import time
  • fixed a bug in 'if file is missing, remove record' file maintenance job. if none of the files yet scanned had any urls, it could error out since the 'missing and invalid files' directory was yet to be created
  • linux users who seem to have mpv support yet are set to use the native viewer will get a one-time popup note on update this week just to let them know that mpv is stable on linux now and how to give it a go
  • the macOS App now spits out any mpv import errors when you hit _help->about_, albeit with some different text around it
  • I maybe fixed the 'hold shift to not follow a dragged page' tech for some users for whom it did not work, but maybe not
  • thanks to a user, the new website now has a darkmode-compatible hydrus favicon
  • all file import options now expose their new 'destination locations' object in a new button in the UI. you can only set one destination for now ('my files', obviously), but when we have multiple local file services, you will be able to set other/multiple destinations here. if you set 'nothing', the dialog will moan at you and stop you from ok-ing it.
  • I have updated all import queues and other importing objects in the program to pause their file work with appropriate error messages if their file import options ever has a 'nothing' destination (this could potentially happen if future after a service deletion). there are multiple layers of checks here, including at the final database level
  • misc code cleanup

  • client api

  • added 'create_new_file_ids' parameter to the 'file_metadata' call. this governs whether the client should make a new database entry and file_id when you ask about hashes it has never seen before. it defaults to **false**, which is a change on previous behaviour
  • added help talking about this
  • added a unit test to test this
  • added archive timestamp and hash hex sort enum definitions to the 'search_files' client api help
  • client api version is now 31

• version 480

  • file notes and media viewer hover windows

  • file notes are now shown on the media viewer! this is a first version, pretty ugly, and may have font layout bugs for some systems, but it works. they hang just below the top-right hover, both in the canvas background and with their own hover if you mouseover. clicking on any note will open 'edit notes' on that note
  • the duplicate filter's always-on hover _should_ slide out of the way when there are many notes
  • furthermore, I rewrote the backend of hover windows. they are now embedded into the media viewer rather than being separate frameless toolbar windows. this should relieve several problems different users had--for instance, if you click a hover, you now no longer lose focus on the main media viewer window. I hacked some of this to get it to work, but along the way I undid three other hacks, so overall it should be better. please let me know how this works for you!
  • fixed a long time hover window positioning bug where the top-right window would sometimes pop in for a frame the first time you moved the mouse to the top middle before repositioning and hiding itself again
  • removed the 'notes' icon from the top right hover window
  • refactored a bunch of canvas background code

  • client api

  • search_files/get_thumbnail now returns image/jpeg or image/png Content-Type. it _should_ be super fast, but let me know if it lags after 3k thumbs or something
  • you can now ask for CBOR or JSON specifically by using the 'Accept' request header, regardless of your own request Content-Type (issue #1110)
  • if you send or ask for CBOR but it is not available for that client, you now get a new 'Not Acceptable' 406 response (previously it would 500 or 200 but in JSON)
  • updated the help regarding the above and wrote some unit tests to check CBOR/JSON requests and responses
  • client api version is now 30

  • misc

  • added a link to 'Hyshare', at https://github.com/floogulinc/hyshare, to the Client API help. it is a neat way to share galleries with friends, just like the the old 'local booru'
  • building on last week's shift-select improvement, I tweaked it and shift-select and ctrl-select are back to not setting the preview focus. you can ctrl-click a bunch of vids in quick silence again
  • the menu on the 'file log' button is now attached to the downloader page lists and the menu when you right-click on the file log panel. you can now access these actions without having to highlight a big query
  • the same is also true of the search/check log!
  • when you select a new downloader in the gallery download page, the keyboard focus now moves immediately to the query text input box
  • tweaked the zoom locking code in the duplicate filter again. the 'don't lock that way if there is spillover' test, which is meant to stop garbage site banners from being hidden just offscreen, is much more strict. it now only cares about 10% or so spillover, assuming that with a large 'B' the spillover will be obvious. this should improve some odd zoom locking situations where the first pair change was ok and the rest were weird
  • if you exit the client before the first session loads (either it is really huge or a problem breaks/delays your boot) the client will not save any 'last/exit session' (previously, it was saving empty here, requiring inconvenient load from a backup)
  • if you have a really really huge session, the client is now more careful about not booting delayed background tasks like subscriptions until the session is in place
  • on 'migrate database', the thumbnail size estimate now has a min-max range and a tooltip to clarify that it is an estimate
  • fixed a bug in the new 'sort by file hash' pre-sort when applying system:limit

• version 479

  • misc

  • when shift-selecting some thumbnails, you can now reverse the direction of the select and what you just selected will be deselected, basically a full undo (issue #1105)
  • when ctrl-selecting thumbnails, if you add to the selection, the file you click is now focused and always previewed (previously this only happened if there was no focused file already). this is related to the shift-select logic above, but it may be annoying when making a big ctrl-selection of videos etc.. so let me know and I can make this more clever if needed
  • added file sort 'file->hash', which sorts pseudorandomly but repeatably. it sounds not super clever, but it will be useful for certain comparison operations across clients
  • when you hit 'copy->hash' on a file right-click, it now shows the sha256 hash for quick review
  • in the duplicate filter, the zoom locking tech now works better™ when one of the pair is portrait and the other landscape. it now tries to select either width or height to lock both when going AB and BA. it also chooses the 'better' of width or height by choosing the zoom that'll change the size less radically. previously, it could do width on AB and height on BA, which lead to a variety of odd situations. there are probably still some issues here, most likely when one of the files almost exactly fills the whole canvas, so let me know how you get on
  • webps with transparency should now load correct! previously they were going crazy in the transparent area. all webps are scheduled a thumbnail regen this week
  • when import folders run, the count on their progress bar now ignores previous failed and ignored entries. it should always start 0, like 0/100, rather than 20/120 etc...
  • when import folders run, any imports where the status type is set to 'leave the file alone' is now still scanned at the end of a job. if the path does not exist any more, it is removed from the import list
  • fixed a typo bug in the recent delete code cleanup that meant 'delete files after export' after a manual export was only working on the last file in the selection. sorry for the trouble!
  • the delete files dialog now starts with keyboard focus on the action radiobox (it was defaulting to ok button since I added the recent panel disable tech)
  • if a network job has a connection error or serverside bandwidth block and then waits before retrying, it now checks if all network jobs have just been paused and will not reattempt the connection if so (issue #1095)
  • fixed a bug in thumbnail fallback rendering
  • fixed another problem with cloudscraper's new method names. it should work for users still on an old version
  • wrote a little 'extract version' sql and bat file for the db folder that simply pull the version from the client.db file in the same directory. I removed the extract options/subscriptions sql scripts since they are super old and out of date, but this general system may return in future

  • file history chart

  • added 'archive' line to the file history chart. this isn't exactly (current_count - inbox_count), but it pretty much is
  • added a 'show deleted' checkbox to the file history chart. it will recalculate the y axis range on click, so if you have loads of deleted files, you can now hide them to see current better
  • improved the way data is aggregated in the file history chart. diagonal lines should be reduced during any periods of client import-inactivity, and spikes should show better
  • also bumped the number of steps up to 8,000, so it should look nice maximised on a 4k
  • the file history chart now remembers its last size and position--it has an entry under options->gui

  • client api

  • thanks to a user, the Client API now accepts any file_id, file_ids, hash, or hashes as arguments in any place where you need to specify a file or files
  • like 'return_hashes', the 'search_files' command in the Client API now takes an optional 'return_file_ids' parameter, default true, to turn off the file ids if you only want hashes
  • added 'only_return_basic_information' parameter, default false, to 'get_metadata' call, which is fast for first-time requests (it is slim but not well cached) and just delivers the basics like resolution and file size
  • added unit tests and updated the help to reflect the above
  • client api version is now 29

  • help

  • split up the 'more files' help section into 'powerful searching' and 'exporting files', both still under the 'next steps' section
  • moved the semi-advanced 'OR' section from 'tags' to 'searching'
  • brushed up misc help
  • a couple of users added some misc help updates too, thank you!

  • misc boring cleanup

  • cleaned up an old wx label patch
  • cleaned up an old wx system colour patch
  • cleaned up some misc initialisation code

• version 478

  • misc

  • if a file note text is crazy and can't be displayed, this is now handled and the best visual approximation is displayed (and saved back on ok) instead
  • fixed an error in the cloudflare problem detection calls for the newer versions of cloudscraper (>=1.2.60) while maintaining support for the older versions. fingers crossed, we also shouldn't repeat this specific error if they refactor again

  • file history chart updates

  • fixed the 'inbox' line in file history, which has to be calculated in an odd way and was not counting on file imports adding to the inbox
  • the file history chart now expands its y axis range to show all data even if deleted_files is huge. we'll see how nice this actually is IRL
  • bumped the file history resolution up from 1,000 to 2,000 steps
  • the y axis _should_ now show localised numbers, 5,000 instead of 5000, but the method by which this occurs involves fox tongues and the breath of a slighted widow, so it may just not work for some machines

  • cleanup, mostly file location stuff

  • I believe I have replaced all the remaining surplus static 'my files' references with code compatible with multiple local file services. when I add the capability to create new local file services, there now won't be a problem trying to display thumbnails or generate menu actions etc... if they aren't in 'my files'
  • pulled the autocomplete dropdown file domain button code out to its own class and refactored it and the multiple location context panel to their own file
  • added a 'default file location' option to 'files and trash' page, and a bunch of dialogs (e.g. the search panel when you make a new export folder) and similar now pull it to initialise. for most users this will stay 'my files' forever, but when we hit multiple local file services, it may want to change
  • the file domain override options in 'manage tag display and search' now work on the new location system and support multple file services
  • in downloaders, when highlighting, a database job that does the 'show files' filter (e.g. to include those in trash or not) now works on the new location context system and will handle files that will be imported to places other than my files
  • refactored client api file service parsing
  • refactored client api hashes parsing
  • cleaned a whole heap of misc location code
  • cleaned misc basic code across hydrus and client constant files
  • gave 'you don't want the server' help page a very quick pass

  • client api

  • in prep for multiple local file services, delete_files now takes an optional file_service_key or file_service_name. by default, it now deletes from all appropriate local services, so behaviour is unchanged from before without the parameter if you just want to delete m8
  • undelete files is the same. when we have multiple local file services, an undelete without a file service will undelete to all locations that have a delete record
  • delete_files also now takes an optional 'reason' parameter
  • the 'set_notes' command now checks the type of the notes Object. it obviously has to be string-to-string
  • the 'get_thumbnail' command should now never 404. if you ask for a pdf thumb, it gives the pdf default thumb, and if there is no thumb for whatever reason, you get the hydrus fallback thumbnail. just like in the client itself
  • updated client api help to talk about these
  • updated the unit tests to handle them too
  • did a pass over the client api help to unify indent style and fix other small formatting issues
  • client api version is now 28

• version 477

  • misc

  • the network engine now parses the 'last-modified' response header for raw files. if this time is earlier than any parsed source time, it is used as the source time and saved to the new 'domain modified time' system. this provides decent post time parsing for a bunch of sites by default, which will also help for subscription timing and similar
  • to get better apng duration, updated the apng parser to count up every frame duration separately. previously, if ffmpeg couldn't figure it out, I was just defaulting to 24 fps and estimating. now it is calculated properly, and for variable framerate apngs too. all apngs are scheduled for a metadata regen this week. thanks to the user who submitted some long apngs where this problem was apparent
  • fixed a bug in the network engine filter that figures out url class precedence. url classes with more parameters were being accidentally sorted above those with more path components, which was messing with some url class matching and automatic parser linking
  • improved the message when an url class fails to match because the given url has too few path components
  • fixed a time delta display bug where it could say '2 years, 12 months' and similar, which was due to a rounding issue on 30 day months and the, for example, 362nd day of the year
  • fixed a little bug where if you forced an archive action on an already archived file, that file would appear to get a fake newer archived timestamp in UI until you restarted
  • updated the default nitter parsers to pull a creator tag. this seemed to not have been actually done when previously thought
  • the image renderer now handles certain broken files better, including files truncated to 0 size by disk problem. a proper error popup is made, and file integrity and rescan jobs are scheduled

  • file history chart

  • for a long time, a user has been generating some cool charts on file history (how many files you've had in your db over time, how many were deleted, etc...) in matplotlib. you may have run his script before on your own database. we've been talking a while about integrating it into the client, and this week I finally got around to it and implemented it in QtCharts. please check out the new 'view file history' underneath Mr Bones's entry in the help menu. I would like to do more in this area, and now I have learned a little more about QtCharts I'd like to revisit and polish up my old bandwidth charts and think more about drawing some normal curves and so on of other interesting data. let me know what you think!
  • I did brush up a couple things with the bandwidth bar chart already, improving date display and the y axis label format

  • client api

  • a user has written several expansions for the client api. I really appreciate the work
  • the client api now has note support! there is a new 'add notes' permission, 'include_notes' parameter in 'file_metadata' to fetch notes, and 'set_notes' and 'delete_notes' POST commands
  • the system predicate parser now supports note system preds
  • hydrus now supports bigger GET requests, up to 2 megabytes total length (which will help if you are sending a big json search object via GET)
  • and the client api now supports CBOR as an alternate to JSON, if requested (via content-type header for POST, 'cbor' arg for GET). CBOR is basically a compressed byte-friendly version of JSON that works a bit faster and is more accessible in some lower level languages
  • cbor2 is now in the requirements.txt(s), and about->help shows it too
  • I added a little api help on CBOR
  • I integrated the guy's unit tests for the new notes support into the main hydrus test suite
  • the client api version is now 27
  • I added links to the client api help to this new list of hydrus-related projects on github, which was helpfully compiled by another user: https://github.com/stars/hydrusnetwork/lists/hydrus-related-projects

• version 476

  • domain modified times

  • the downloader now saves the 'source time' (or, if none was parsed, 'creation time') for each file import object to the database when a file import is completed. separate timestamps are tracked for every domain you download from, and a file's number can update to an earlier time if a new one comes in for that domain
  • I overhauled how hydrus stores timestamps in each media object and added these domain timestamps to it. now, when you see 'modified time', it is the minimum of the file modified time and all recorded domain modified times. this aggregated modfified time works for sort in UI and when sorting before applying system:limit, and it also works for system:modified time search. the search may be slow in some situations--let me know
  • I also added the very recent 'archived' timestamps into this new object and added sort for archived time too. 'archived 3 minutes ago' style text will appear in thumbnail right-click menus and the media viewer top status text
  • in future, I will add search for archive time; more display, search, and sort for modified time (for specific domains); and also figure out a dialog so you can manually edit these timestamps in case of problems
  • I also expect to write an optional 'fill in dummy data' routine for the archived timestamps for files archived before I started tracking these timestamps. something like 'for all archived files, put in an archive time 20% between import time and now', but maybe there is a better way of doing it, let me know if you have any ideas. we'll only get one shot at this, so maybe we can do a better estimate with closer analysis
  • in the longer future, I expect import/export support for this data and maintenance routines to retroactively populate the domain data based on hitting up known urls again, so all us long-time users can backfill in nicer post times for all our downloaded files

  • searching tags on client api

  • a user has helped me out by writing autocomplete tag search for the client api, under /add_tags/search_tags. I normally do not accept pull requests like this, but the guy did a great job and I have not been able to fit this in myself despite wanting it a lot
  • I added some bells and whistles--py 3.8 support, tag sorting, filtering results according to any api permissions, and some unit tests
  • at the moment, it searches the 'storage' domain that you see in a manage tags dialog, i.e. without siblings collapsed. I can and will expand it to support more options in future. please give it a go and let me know what you think
  • client api version is now 26

  • misc

  • when you edit something in a multi-column list, I think I have updated every single one so the selection is preserved through the edit. annoyingly and confusingly on most of the old lists, for instance subscriptions, the 'ghost' of the selection focus would bump up one position after an edit. now it should stay the same even if you rename etc... and if you have multiple selected/edited
  • I _think_ I fixed a bug in the selected files taglist where, in some combination of changing the tag service of the page and then loading up a favourite search, the taglist could get stuck on the previous tag domain. typically this would look as if the page's taglist had nothing in it no matter what files were selected
  • if you set some files as 'alternates' when they are already 'duplicates', this now works (previously it did nothing). the non-kings of the group will be extracted from the duplicate group and applied as new alts
  • added a 'BUGFIX' checkbox to 'gui pages' options page that forces a 'hide page' signal to the current page when creating a new page. we'll see if this patches a weird error or if more work is needed
  • added some protections against viewing files when the image/video file has (incorrectly) 0 width or height
  • added support for viewing non-image/video files in the duplicate filter. there are advanced ways to get unusual files in here, and until now a pdf or something would throw an error about having 0 width

• version 475

  • new help docs

  • the hydrus help is now built from markup using MkDocs! it now looks nicer and has search and automatically generated tables of contents and so on. please check it out. a user converted _all_ my old handwritten html to markup and figured out a migration process. thank you very much to this user.
  • the help has pretty much the same structure, but online it has moved up a directory from https://hydrusnetwork.github.io/hydrus/help to https://hydrusnetwork.github.io/hydrus. all the old links should redirect in any case, so it isn't a big deal, but I have updated the various places in the program and my social media that have direct links. let me know if you have any trouble
  • if you run from source and want a local copy of the help, you can build your own as here: https://hydrusnetwork.github.io/hydrus/about_docs.html . it is super simple, it just takes one extra step. Or just download and extract one of the archive builds
  • if you run from source, hit _help->open help_, and don't have help built, the client now gives you a dialog to open the online help or see the guide to build your help
  • the help got another round of updates in the second week, some fixed URLs and things and the start of the integration of the 'simple help' written by a user
  • I added a screenshot and a bit more text to the 'backing up' help to show how to set up FreeFileSync for a good simple backup
  • I added a list of some quick links back in to the main index page of the help
  • I wrote an unlinked 'after_distaster' page for the help that collects my 'ok we finished recovering your broken database, now use your pain to maintain a backup in future' spiel, which I will point people to in future

  • misc

  • fixed a bug where changes to the search space in a duplicate filter page were not sticking after the first time they were changed. this was related to a recent 'does page have changes?' optimisation--it was giving a false negative for this page type (issue #1079)
  • fixed a bug when searching for both 'media' and 'preview' view count/viewtime simultaneously (issue #1089, issue #1090)
  • added support for audio-only mp4 files. these would previously generally fail, sometimes be read as m4a. all m4as are scheduled for a metadata regen scan
  • improved some mpeg-4 container parsing to better differentiate these types
  • now we have great apng detection, all pngs with apparent 'bitrate' over 0.85 bits/pixel will be scheduled for an 'is this actually an apng?' scan. this 0.85 isn't a perfect number and won't find extremely well-compressed pixel apngs, but it covers a good amount without causing a metadata regen for every png we own
  • system:hash now supports 'is' and 'is not', if you want to, say, exclude a list of hashes from a search
  • fixed some 'is not' parsing in the system predicate parser
  • when you drag and drop a thumbnail to export it from the program, the preview media viewer now pauses that file (just as the full media viewer does) rather than clears it
  • when you change the page away while previewing media with duration, the client now remembers if you were paused or playing and restores that state when you return to that page
  • folded in a new and improved Deviant Art page parser written by a user. it should be better about getting the highest quality image in unusual situations
  • running a search with a large file pool and multiple negated tags, negated namespaces, and/or negated wildcards should be significantly faster. an optimisation that was previously repeated for each negated tag search is now performed for all of them as a group with a little inter-job overhead added. should make '(big) system:inbox -character x, -character y, -character z' like lightning compared to before
  • added a 'unless namespace is a number' to 'tag presentation' options, which will show the full tag for tags like '16:9' when you have 'show namespaces' unticked
  • altered a path normalisation check when you add a file or thumbnail location in 'migrate database'--if it fails to normalise symlinks, it now just gives a warning and lets you continue. fingers crossed, this permits rclone mounts for file storage (issue #1084)
  • when a 'check for missing/invalid file' maintenance job runs, it now prints all the hashes of missing or invalid files to a nice simple newline-separated list .txt in the error directory. this is an easy to work with hash record, useful for later recovery
  • fixed numerous instances where logs and texts I was writing could create too many newline characters on Windows. it was confusing some reader software and showing as double-spaced taglists and similar for exported sidecar files and profile logs
  • I think I fixed a bug, when crawling for file paths, where on Windows some network file paths were being detected incorrectly as directories and causing parse errors
  • fixed a broken command in the release build so the windows installer executable should correctly get 'v475' as its version metadata (previously this was blank), which should help some software managers that use this info to decide to do updates (issue #1071)

  • some cleanup

  • replaced last instances of EVT_CLOSE wx wrapper with proper Qt code
  • did a heap of very minor code cleanup jobs all across the program, mostly just to get into pycharm
  • clarified the help text in _options->external programs_ regarding %path% variable

  • pycharm

  • as a side note, I finally moved from my jank old WingIDE IDE to PyCharm in this release. I am overall happy with it--it is clearly very powerful and customisable--but adjusting after about ten or twelve years of Wing was a bit awkward. I am very much a person of habit, and it will take me a little while to get fully used to the new shortcuts and UI and so on, but PyCharm does everything that is critical for me, supports many modern coding concepts, and will work well as we move to python 3.9 and beyond

• version 474

  • command palette

  • the guy who put the command pallete together has fixed a 'show palette' bug some people encountered (issue #1060)
  • he also added mouse support!
  • he added support to show checkable menu items too, and I integrated this for the menubar (lightning bolt icon) items
  • I added a line to the default QSS that I think fixes the odd icon/text background colours some users saw in the command palette

  • misc

  • file archive times are now recorded in the background. there's no load/search/sort yet, but this will be added in future
  • under 'manage shortcuts', there is a new checkbox to rename left- and right-click to primary- and secondary- in the shortcuts UI. if you have a flipped mouse or any other odd situation, try it out
  • if a file storage location does not have enough free disk space for a file, or if it just has <100MB generally, the client now throws up a popup to say what happened specifically with instructions to shut down and fix now and automatically pauses subscriptions, paged file import queues, and import folders. this test occurs before the attempt to copy the file into place. free space isn't actually checked over and over, it is cached for up to an hour depending on the last free space amount
  • this 'paused all regular imports' mode is also now fired any time any simple file-add action fails to copy. at this stage, we are talking 'device disconnected' and 'device failed' style errors, so might as well pause everything just to be careful
  • when the downloader hits a post url that spawns several subsidiary downloads (for instance on pixiv and artstation when you have a multi-file post), the status of that parent post is now 'completed', a new status to represent 'good, but not direct file'. new download queues will then present '3N' and '3 successful' summary counts that actually correspond to number of files rather than number of successful items
  • pages now give a concise 'summary name' of 'name - num_files - import progress' (it also eli...des for longer names) for menus and the new command palette, which unlike the older status-bar-based strings are always available and will stop clients with many pages becoming multi-wide-column-menu-hell
  • improved apng parsing. hydrus can now detect that pngs are actually apngs for (hopefully) all types of valid apng. it turns out some weird apngs have some additional header data, but I wrote a new chunk parser that should figure it all out
  • with luck, users who have window focus issues when closing a child window (e.g. close review services, the main gui does not get focus back), should now see that happen (issue #1063). this may need some more work, so let me know
  • the session weight count in the 'pages' menu now updates on any add thumbs, remove thumbs, or thumbnail panel swap. this _should_ be fast all the time, and buffer nicely if it is ever overwhelmed, but let me know if you have a madlad session and get significant new lag when you watch a downloader bring in new files
  • a user came up with a clever idea to efficiently target regenerations for the recent fix to pixel duplicate calculations for images with opaque alpha channels, so this week I will queue up some pixel hash regeneration. it does not fix every file with an opaque alpha channel, but it should help out. it also shouldn't take _all_ that long to clear this queue out. lastly, I renamed that file maintenance job from 'calculate file pixel hash' to 'regenerate pixel duplicate data'
  • the various duplicate system actions on thumbnails now specify the number of files being acted on in the yes/no dialog
  • fixed a bug when searching in complicated multi-file-service domains on a client that has been on for a long time (some data used here was being reset in regular db maintenance)
  • fixed a bug where for very unlucky byte sizes, for instance 188213746, the client was flipping between two different output values (e.g. 179MB/180MB) on subsequent calls (issue #1068)
  • after some user profiles and experimental testing, rebalanced some optimisations in sibling and parent calculation. fingers crossed, some larger sibling groups with worst-case numbers should calculate more efficiently
  • if sibling/parent calculation hits a heavy bump and takes a really long time to do a job during 'normal' time, the whole system now takes a much longer break (half an hour) before continuing

  • boring stuff

  • the delete dialog has basic multiple local file service support ready for that expansion. it no longer refers to the old static 'my files' service identifier. I think it will need some user-friendly more polish once that feature is in
  • the 'migrate tags' dialog's file service filtering now supports n local file services, and 'all local files'
  • updated the build scripts to force windows server 2019 (and macos-11). github is rolling out windows 2022 as the new latest, and there's a couple of things to iron out first on our end. this is probably going to happen this year though, along with Qt6 and python 3.9, which will all mean end of life for windows 7 in our built hydrus release
  • removed the spare platform-specific github workflow scripts from the static folder--I wanted these as a sort of backup, but they never proved useful and needed to be synced on all changes

• version 473

  • misc

  • fixed the recent problem with drag and dropping thumbnails to a level below the top row of pages. sorry for the trouble!
  • fixed a bug where the client would not load results sorting by 'import time' when the search file domain was a single deleted file domain
  • fixed a list display bug in the edit page parser dialog when a subsidiary page parser has two complicated string-match based content parsers
  • collections now sort by modified time, using the largest known modified time in their collection
  • added sqlite3.exe console back into the windows build--sorry, it was missing since the github build changeover!
  • added a note to the help about backing up when tight on space, which I will repeat here: the sqlite database files are very compressible (70GB->17GB on default 7zip settings!), so if you need more space on your backup drive, this is a good way to reclaim it

  • command palette

  • a user has written a cool 'command palette' for the program! it brings up a type-and-search interface to navigate to pages or menu entries.
  • I have integrated his first version and set the default shortcut to Ctrl+P. users who update will get this shortcut if they have nothing else on Ctrl+P on 'main window' set. if you prefer Ctrl+K or anything else, you can change it under _file->shortcuts->the main window_
  • regular users will get a page list they can search and select, advanced users will also get the (potentially dangerous) full scan of the menubar and current thumbnail right-click menu. I will be polishing this latter feature in future to filter out big maintenance jobs and show checkbox status and similar, so if you are advanced, please be careful for now
  • try it out, and let me know how it goes. the underlying widget is neat, and I can change its behaviour and extend it significantly

  • (mostly advanced) deleted file improvements

  • files that have been deleted from a local file domain are now aware of their file deletion reason. this is visible in the right-click menu of thumb or media canvas
  • the advanced file deletion dialog now initialises using this stored reason. if all pending deletees have the same existing reason stored, it will display it, and if they are all set but differ, this will be noted and an option to not alter them is also available. this will come up later in niche advanced situations with mutiple file services
  • reversing a recent change, local file deletion reasons are no longer cleared on undelete or (re)import. they'll now hang around invisibly and initialise any future advanced file deletion dialog
  • updated the thumbnail and canvas undelete mechanism to handle multiple services. now, if the files are deleted in more than one domain, you will be asked to multiple-select which you wish to undelete for. if there is only one eligible undelete service, the process remains unchanged--you'll just get a yes/no confirmation if the 'confirm trash' option is set
  • misc multiple local file services code conversion work

• version 472

  • highlights

  • the file domain button of every autocomplete input now has a 'multiple locations' entry. this launches a checkboxlist of all possible search locations and allows you to search more than one domain at once. it works, too! in future, when we can have multiple 'my files' services, you'll be able to choose here unions of what to search. users in advanced mode will see repository updates, all local files, all known files, and the new deleted file domains on this list. I removed the deleted file domains from the front menu because I expect them to be rarely used
  • in _options->thumbnails_, there is now a 'thumbnail scaling' dropdown. you can set it so thumbs only ever scale down (which remains the default), scale to fit (i.e. very small images are also scaled up), or scale to fill. the 'animation' as thumbnails refit and delayed-regen themselves to 'scale to fill' is accidentally one of the coolest things I have done
  • removed the old 'EXPERIMENTAL: thumbnail fill' option. the new mode works essentially the same, but faster and higher quality
  • in the page tab menu, there is a new submenu 'pages', which shows all the pages at or below the current level. if you right-click on a page of pages tab, it will just show for that page of pages. click any of the entries, you will select that page. it is a web browser-like quick navigation menu, let me know what you think!
  • rejiggered the page tab menu a little, reordering groups a bit with nicer separators and putting 'select' navigation on the menu even if you click in greyspace
  • fixed a problem in page tab menu logic where if you right-clicked on greyspace, it would render the menu for the bottommost page of pages row rather than the one actually clicked
  • last week's update where a mouse release event will no longer fire in the shortcuts system if the mouse moved a decent distance between press and release should now work in the media viewer canvas when dragging is set to anchor the mouse in place. some advanced users may wish to try setting archive/delete to work on mouse release and use left click to drag

  • bug fixes

  • fixed pages force-refreshing file queries on session load. this has never been intentional, but it slipped through again and was happening for a month or two now. I have added an explicit test to my routine to make sure this doesn't happen again, sorry for the trouble!
  • fixed a problem in the recent fast shutdown code that was accidentally also shutting down some maintenance work like repository processing as soon as it started, even if 'exit and force work' was chosen
  • all images with a completely opaque alpha channel will now have that alpha channel dropped for the new pixel hash calculation, meaning they will now match with regular non-alpha images with the same colour pixel data. in fact, all images with an opaque alpha now have that channel dropped on load, which will save a little memory and CPU any time they are handled (issue #770)
  • if the 'durable' temporary database exists on boot, it is now deleted and a fresh one created rather than trying to re-use the old one (which would not have any useful information anyway), and a note is made to log. one user recently had a problem where an existing corrupt temp dir was stopping boot, which this fixes

  • misc

  • updated the windows build to use sqlite 3.37.2, the sqlite3 in the db dir is also updated
  • the deleted files system now neatly cleans up old file deletion reasons on file import and file undelete
  • cleared out some old thumbnail generation code, including deleting an old and now obselete optimisation where too-large thumbs were scaled down to make new thumbs rather than revisiting source. since our thumb situation is more complicated now, this is gone in favour of nicer quality thumbs and simpler code
  • fixed up some upcoming database maintenance code in my new modules
  • updated and cleaned the code in the old wx-converted checkboxlist and replaced some awkward old access routines
  • cleared out some old HTA archive code

• version 471

  • times

  • if you have file viewing stats turned on (by default it is), the client will now track the 'last viewed time' of your files, both in preview and media viewers. a record is only made assuming they pass the viewtime checks under _options->file viewing statistics_ (so if you scroll through really quick but have it set to only record after five seconds of viewing, it will not save that as the last viewed time). this last viewed time is shown on the right-click menu with the normal file viewing statistics
  • sorting by 'import time' and 'modified time' are moved to a new 'time' subgroup in the sort button menu
  • also added to 'time' is 'last viewed time'. note that this has not been tracked until now, so you will have to look at a bunch of things for a few seconds each to get some data to sort with
  • to go with 'x time' pattern, 'time imported' is renamed to 'import time' across the program. both should work for system predicate parsing
  • system:'import time' and 'modified time' are now bundled into a new 'system:time' stub in the system predicates list. the window launched from here is an experimental new paged panel. I am not sure I really like it, but let's see how it works IRL
  • 'system:last view time' is added to search the new field! give it a go once you have some data
  • also note that the search and sort of last viewed time works on the 'media viewer' number. those users who use preview or combined numbers for stuff, let me know if and how you would like that to work here--sort/search for both media and preview, try to combine based on the logic in the options, or something else?

  • loading serialised pngs

  • the client can now load serialised downloader-pngs if they are a perfect RGB conversion of an original greyscale export.
  • the pngs don't technically have to be pngs anymore! if you drag and drop an image from firefox, the temporary bitmap exported and attached to the DnD _should_ work!
  • the lain easy downloader import now has a clipboard paste button. it can take regular json text, and now, bitmap data!
  • the 'import->from clipboard' button action in many multiple column lists across the program (e.g. manage parsers) (but not every list, a couple are working on older code) also now accepts bitmap data on the clipboard
  • the various load errors here are also improved

  • custom widget colors

  • (advanced users only for now)
  • after banging my head against it, I finally figured out an ok way to send colors from a QSS style file to python code. this means I can convert my custom widgets to inherit colours from the current QSS. I expect to migrate pretty much everything currently fixed over to this, except tag colours and maybe some thumbnail border stuff, and retire the old darkmode
  • if you are a QSS lad, please check out the new entries at the bottom of default_hydrus.qss and play around with them in your own QSSes. please do not send me any updates to be folded in to the install yet as I still have a bunch of other colours to add. this week is just a test--please let me know how it works for you

  • misc

  • mouse release events no longer trigger a command in the shortcuts system if the release happens more than about 20 pixels from the original mouse down. this is tricky, so if you are into clever shortcuts, let me know how it works for you
  • the file maintenance manager (which has been getting a lot of work recently with icc profiles, pixel dupes, some thumb regen, and new audio channel checks), now saves its work and publishes updates faster to the UI, at least once every ten seconds
  • the sort entries in the page sort control are now always sorted according to their full (type, name) string, and the mouse-wheel-to-navigate is now fixed to always mirror this
  • improved some 'delete file reason' handling. currently, a file deletion reason should only be applied when a file is entering trash. there was a bug that force-physical-deleting files from trash would overwrite their original deletion reason. this is now fixed. the advanced delete files dialog now disables the whole reason panel better when needed, never sends a file reason down to the database when there should be no reason, disables the panel if all the files are in the trash, and at the database level when file deletion reasons are being set, all files are filtered for status beforehand to ensure none are accidentally set by other means. I am about to make trash more intelligent as part of multiple local file services, so I expect to revisit this soon
  • the new ICC Profile conversion no longer occurs on I or F mode files. there are weird 32/64 bit monochrome files, and mode/ICC conversion goes whack with current PIL code
  • replaced the critical hamming test in the duplicate files system with a different bit-counting strategy that works about 9% faster. hamming test is used in all duplicate file searching, so this should help out a tiny bit in a lot of places

  • boring cleanup

  • cleaned up how media viewer canvas type is stored and tested in many places
  • all across the program, file viewing statistics are now tracked by type rather than a hardcoded double of preview & media viewer. it will take a moment to update the database to reflect this this week
  • cleaned up a ton of file viewing stats code
  • cleared out the last twenty or so uses of the old 'execute many select' database access routine in favour of the new lower-overhead and more query-optimisable temporary integer tables method

• version 470

  • multiple file services

  • I finished the conversion of all UI search to the new multiple location object. everything from back- to frontend now supports cleverer search. since searching deleted files is simple to add, users in advanced mode will now see 'deleted from...' in a new list in the tag autocomplete dropdown file domain button
  • the next step is writing a widget that allows multiple selection, and then all this should work right out of the box, and we'll be an important step closer to allowing multiple local file services

  • misc

  • the video parsing routine is better at detecting when a present audio track is actually silent (and hence when it should mark a video as 'no audio'). all video with audio will be requeued for a metadata reparse in the files maintenance system on update
  • fixed an error from last week when trying to create a new page from the tags (e.g. middle-clicking them) in the active search list
  • added 'audio mkv' format to the client, to represent mkvs without a video track. I think most of the time this is going to be audio track webms from youtube-dl and similar
  • added 'file relationships: set files as potential duplicates' command to the 'media actions' shortcut set
  • I expanded the 'backing up' section in 'installing and updating' help
  • I wrote an 'anti-virus' section for 'installing and updating' help, since I kept writing the same basic spiel about false positives. please feel free to point people there in future to relieve their concerns
  • improved some shutdown tests, the client and server should exit faster in some cases (e.g. when a hydrus repository network job is hanging on reconnection attempts, holding up the 'synchronise_repository' daemon shutdown)
  • the 'file was xxx at (y timestamp), which was (z time units) before this check' line in file import notes now always puts 'z time units' as that, ignoring the 'always show ISO time' setting, which was just substituting it with 'y timestamp' again. let me know if you spot other bad grammar with this setting on, I'll fix it!
  • fingers crossed, images in the LAB colourspace _should_ now normalise to sRGB with the correct whitepoint. thanks to the user who provided example test tiff images here. this now uses the new PIL-based colourspace conversion I used to make ICC profiles work, just on LAB->sRGB. as far as I understand, OpenCV uses a fixed whitepoint of D65, resulting in yellow/warm conversions for some formats, but PIL may be able to figure out if D50 is needed??? if you have some crazy LUV or YPbPr or YIQ image that shows up wrong, please send it in and I'll see what I can do!

  • boring rewrites and cleanup while doing file service work

  • many more UI objects now store and do file service logic using a more complicated 'location context', which can store a mix of multiple services and 'deleted from service' data. all the search code that works on this can now propagate to display

  • the management objects behind every page now store a multiple location object, not a single file service id
  • all media panels (the thumbnail grid on a page) are now instantiated by a multiple location object, and when they serve a highlighted downloader, they now inherit that from the file import options, which in future will dictate import destinations
  • all canvases are now the same, inheriting their new location context from their parents
  • all tag lists are the same. mostly they don't care much about file domain, but when you middle-click to create new pages from the autocomplete dropdown list or active search list, it can matter, so they now propagate it along
  • the underlying medialist objects are now the same, and various delete logic (e.g. 'should we remove this thumb we just deleted?') is updated to work on complex domains
  • some duplicate lookup code now works on location context
  • renamed 'location search context' object to 'location context' since it is used all over now and put it in its own file. also wrote it some neater initialisation and meta object code
  • mr bones now gives duplicate data based on the union of all non-trash local services sans update files (another case of now supporting n services but n is fixed for the moment at 1, 'my files')
  • a bunch of places across the program that used to default to 'my files' or 'all local files' (which is everything on disk, including trash and repository update files) now default to this new union of all non-trash local media services
  • when doing page-to-page file drag and drops, the location context is now preserved (previously, the new page would always be 'my files')
  • whole heap of other cleanup in these systems
  • when a thumbnail cannot be provided (for deleted files or many 'all known files' situations), the thumbnail cache now provides the hydrus icon stand-in instantly, no delayed waterfall
  • fixed an unusual situation where the file search could not provide a file in a tagless search when that file had no detailed file info row in the database. this seems to effect a legacy borked row or two in the new deleted file domain searches
  • removed some ancient dumper status code from thumbnail objects

• version 469

  • misc

  • the 'search log' button and the window panel now let you delete log entries. you can delete by completion status from the menu or specifically by row in the panel (just like the file log)
  • fixed the new 'file is writable' checks for Linux/macOS, which were testing permissions overbroadly and messing with users with user-only permissions set. the code now hands off specific user/group negotiation to the OS. thanks to the maintainer of the AUR package for helping me out here (issue #1042)
  • the various places where a file's permission bits are set are also cleaned up--hydrus now makes a distinction between double-checking a file is set user-writable before deleting/overwriting vs making a file's permission bits (which were potentially messed up in the past) 'nice' for human use after export. in the latter case, which still defaults to 644 on linux/macOS, the user's umask is now applied, so it should be 600 if you prefer that
  • fixed a bug where the media viewer could have trouble initialising video when the player window instantiation was delayed (e.g. with embed button)

  • client api

  • added 'return_hashes' boolean parameter to GET /get_files/search_files, which makes the command return hashes instead of file ids. this is documented in the help and has a new unit test
  • client api version is now 25

  • multiple local file services work

  • I rewrote a lot of code this week, but it proved more complex than I expected. I also discovered I'll have to switch the pages and canvases over too before I can nicely switch the top level UI over to allow multiple search. rather than release a borked feature, I decided not to rush the final phase, so this remains boring for now! the good news is that it works well when I hack it in, so I just need to keep pushing
  • rewrote the caller side of tag autocomplete lookup to work on the new multiple file search domain
  • rewrote the main database level tag lookup code to work on the new multiple file search domain
  • certain types of complicated tag autocomplete lookup, particularly on all known tags and any client with lots of siblings, will be faster now
  • an unusual and complicated too-expansive sibling lookup on autocomplete lookups on 'all known tags' is now fixed

  • boring cleanup and refactoring

  • predicate counts are now managed by a new object. predicates also support 0 minimum count for x-y count ranges, which is now possible when fetching count results from non-cross-referenced file domains (for now this means searching deleted files)
  • cleaned up a ton of predicate instantiation code
  • updated autocomplete, predicate, and pred count unit tests to handle the new objects and bug fixes
  • wrote new classess to cover convenient multiple file search domain at the database level and updated a bunch of tag autocomplete search code to use it
  • misc cleanup and refactoring for file domain search code
  • purged more single file service inspection code from file search systems
  • refactored most duplicate files storage code (about 70KB) to a new client db module

• version 468

  • misc

  • fixed an issue where the one pixel border on the new 'control bar' on the media viewer was annoyingly catching mouse events at the bottom of the screen when in borderless fullscreen mode (and hence dragging the video, not scanning video position). the animation scanbar now owns its own border and processes mouse events on it properly
  • fixed a typo bug in the new pixel hash system that meant new imports were not being added to the system correctly. on update, all files affected will be fixed of bad data and scheduled for a pixel hash regen. sorry for the trouble, and thank you for the quick reports here
  • added a 'fixed font size example' qss file to the install. I have passed this file to others as an example of a quick way to make the font (and essentially ui scale) larger. it has some help comments inside and is now always available. the default example font size is lmao
  • fixed another type checking problem for (mostly Arch/AUR) PyQt5 users (issue #1033)
  • wrote a new display mappings maintenance routine for the database menu that repopulates the display mappings cache for missing rows. this task will be radically faster than a full regen for some problems, but it only solves those problems
  • on boot, the program now explicitly checks if any of the database files are set as read-only and if so will dump out with an appropriate error
  • rewrote my various 'file size problem' exception hierarchy to clearly split 'the file import options disallow this big gif' vs 'this file is zero size/this file is malformed'. we've had several problems here recently, but file import options rule-breaking should set 'ignore' again, and import objects should be better about ignore vs fail state from now on
  • added more error handling for broken image files. some will report cleaner errors, some will now import
  • the new parsing system that discards source urls if they share a domain with a primary import url is now stricter--now discarding only if they share the same url class. the domain check was messing up saving post urls when they were parsed from an api url (issue #1036)
  • the network engine no longer sends a Range header if it is expecting to pull html/json, just files. this fixes fetching pages from nijie.info (and several other server engines, it seems), which has some unusual access rules regarding Range and Accept-Encoding
  • fixed a problem with no_daemons and the docker package server scripts (issue #1039)
  • if the server engine (serverside or client api) is running a request during program shutdown, it now politely says 'Application is shutting down!' with a 503 rather than going bananas and dumping to log with an uncaught 500
  • fixed some bad client db update error handling code

  • multiple local file services (system predicate edition)

  • system:file service now supports 'deleted' and 'petitioned' status
  • advanced 'all known files' search pages now show more system predicates
  • when inbox and archive are hidden because one has 0 count, and the search space is simple, system everything now says what they are, e.g. "system:everything (24) (all in inbox)"
  • file repos' 'system:local/not local' now sort at the top of the system predicate list, like inbox/archive

  • client api

  • the GET /get_files/file_metadata call now returns the file modified date and imported/deleted timestamps for each file service the file is currently in or deleted from. check the help for an example!
  • fixed client api file search with random sort (file_sort_type = 4)
  • client api version is now 24

  • boring multiple local file services work

  • the system predicates listed in a search page dropdown now support the new 'multiple location search context' object, which means in future I will be able to switch over to 'file domain is union of (A, deleted from B, and C)' and all the numbers will add up appropriately with ranged 'x-y' counts and deal with combinations of file repo and local service and current/deleted neatly
  • when fetching system preds in 'all known files', the system:everything 'num files' count will be stated if available in the cache
  • for the new system:file service search, refactored db level file filtering to support all status types
  • cleaned up how system preds are generated

  • boring refactoring work

  • moved GUGs from network domain code to their own file
  • moved URL Class from network domain code to its own file
  • moved the pure functions from network domain code to their own file
  • cleared up some file maintenance enum variable names
  • sped up random file sort for large result sets
  • misc client network code cleanup and type hints, and rejiggered cleaner imports after the refactoring

• version 467

  • new scanbar cleanup

  • the media container's scanbar and volume control are now combined on the same widget, meaning they now show/hide in sync and faster. their layout calculation is also more sensible. the new controls bar also has a thin border to make it pop better against a background video
  • improved the way some auto-hide anti-flicker tech on the scanbar now works. it all hides a frame faster sometimes
  • figured out some new anti-flicker tech to reduce/eliminate a frame of stretch when flicking from a static image to an mpv video, particularly for the first or second time in a session
  • fixed a bug where clicking the global mute/unmute on an mpv player meant that certain shortcut keys (usually those with arrow keys) would not work on that player again. (it was a focus issue on the button, which then captured some form navigation keys but they had nowhere to go)
  • brushed up some mouse coordinate testing logic across the program. some linux clients had trouble with the new animation scanbar popping up over mpv, I think I improved it!
  • fixed another type problem with newer python/PyQt5 on Arch, also in scanbar coordinate testing
  • fixed some dodgy colours in the scanbar initialisation and volume control border
  • macOS users: I undid a long-time paint hack on the media container and the static image canvas. Qt is responsible for clearing the background again, which allows me to remove some jank anti-flicker tech. HOWEVER, the original reason for this hack was because without it, old macOS went to 100% CPU whenever the media viewer was showing something. therefore, to be safe, this option is still on for macOS users for now. you'll get a little flicker when browsing. please try hitting _help->debug->gui actions->macOS anti-flicker test_ and do some mixed video/image browsing. does your whole damn client lock up?

  • misc

  • the 'file log' and 'search log' buttons are now a new widget class that puts an arrow on the side that opens a menu. the secret right-click menus of these buttons is now exposed for all
  • fixed a bug affecting some greyscale pngs with ICC profiles--they were coming out pure white due to a colourspace conversion problem
  • fixed an import problem when PIL could not load a file (due to file malformation) but OpenCV could. this was causing a failed import from the new ICC profile detection code
  • when the downloader hits a broken image file that cannot be imported due to malformation, the status is now 'error' instead of the incorrect 'ignored'
  • fixed the duplicate file filesize comparison statement sometimes showing > in one direction and ≈ in the other. it happened when the larger file was between 20/19 and 21/20 times the size of the smaller, just a logic typo (issue #1028)
  • the trash maintenance daemon is moved from the old threaded daemon system to the new repeating job worker pool. this is the last daemon cleaned up, so I am retiring the old and mostly defunct 'no_daemons' launch argument. a variety of other daemon infrastructure for things like shutdown checks is similarly removed. the program also now waits for the newer daemon jobs to finish working on shutdown
  • moved most client daemon jobs like repository sync and dirty object save down so they start after the first session is loaded rather than right after boot
  • if a file is called to regen its thumbnail but currently has no dimension, this is now a no-op rather than an error. in the situation where users force thumb regen before metadata regen and encounter this, it is sorted out later when the metadata regen recognises new dimensions and reschedules the thumb regen
  • added an extensive user-written guide to the --db_synchronous_override launch argument to the launch arguments help page. it is possible and safe to run the program with synchronous=0 as long as certain caveats are obeyed. thanks to the user who figured this out and wrote it up
  • the downloader engine now discards source urls in an import job if they have the same domain as any existing primary url. this will ensure that if a booru has a link back to itself as a source url, when the 'source' is really an alternate rather than a dupe, it won't be added in hydrus as a known url for that imported file
  • misc cleanup in downloader system and file/search log UI
  • fixed a type bug in the file and search log 'import from png' action. if you have existing pngs previously exported from here, they will import ok now
  • refactored the various hydrus compression code to a new HydrusCompression file
  • exported serialisable data pngs such as from file or search log that hold simple Strings now always compress the data before embedding it in the png. existing pngs that hold uncompressed strings should still load ok
  • the payload in an exported png is now always compressed, and the payload description always states the uncompressed size
  • sped up client shutdown when network traffic has been paused the whole time and a repo sync job might have wanted to run. these jobs also do not hang on a thread worker if network traffic is paused, but they should wake immediately when it is unpaused
  • the hydrus login system is now resistant to connection failures; previously it was getting hung up and jamming the whole hydrus sync system when a server was down

  • client api

  • added GET /manage_database/mr_bones to the Client API. it returns a JSON Object with the same numbers used in the _help->how boned am I?_ dialog
  • incremented Client API version to 23

• version 466

  • video scanbar autohide

  • the scanbar that shows below audio and video is now embedded inside the video frame, and it show/hides based on how close your mouse is to it
  • I've wanted to do this for a long time, since it will allow you to watch 16:9 videos at true 100% in borderless fullscreen, but the hackery of how the media viewer works behind the scenes means this took more work than you'd think and is still a little jank. there's a small amount of flicker when it pops in and out, which I will work on in future. in any case, please have a play with it and let me know what you think. I expect to add some more options, like for the activation padding area around it, and I will be tidying up more layout stuff throughout the media viewer
  • if you are a mostly keyboard user, please check out the new 'global' shortcut to flip on/off a 'force the animation scanbar to show' mode
  • I don't really want to bring back the always-on hanging-below scanbar that just takes up space, but if you try this new embedded scanbar and really hate it, we'll see what we can do

  • more duplicate filter search options

  • the duplicates page now has a dropdown on the search for 'must be/can be/excludes pixel dupes'!
  • the duplicates page now has a number control on the search for what distance the pair was found at! I am not sure how accurate this thing is in all cases, but it seems I started tracking this data some time ago and forgot I even had it
  • these new options are remembered in your session and _should_ remain fast in most normal cases. I put time into some complicated database work this week to get this going, please let me know if you have any trouble with it

  • misc

  • when the export filename pattern in the export files dialog means many of the files share the same base and hence need to do 'filename (5)'-style suffixes to be unique, the number here is now calculated much more efficiently. opening this dialog on 10,000 files with an oft-duplicate pattern should now be a jolt of lag but not many minutes
  • when you choose to 'separate' a subscription with more than 100 queries, you are no longer forced to break it into half
  • when you do break a subscription in half, it now makes sure to sort the query texts before separating
  • if you are in advanced mode, the 'selection tags' list on the left of every page can now switch its tag display type between 'multiple media views', 'display', and 'storage'. this is experimental and a bunch of stuff like 'select files with this tag' won't work yet
  • janitors' petition pages now start with their tag list in 'storage' mode, so you can see the actual tags being changed rather than with siblings and parents calculated
  • rebalanced some janitor mapping petition weights. jannies _should_ see a smoother balance of 'lots of small petitions' vs 'a few larger petitions' amongst petitions all with the same reason and creator

  • boring cleanup and little fixes

  • when you set the checker options in the edit subscription dialog, the queries now recalculate their file velocity better. previously, they would just set 'unknown' and recalc on the next run, but now they will actually recalculate if the query container is loaded into memory or otherwise put a status that says 'will calculate on next run'
  • removed the 'should be namespaced' reason from the manage tags quick petition reasons. this is now all handled by siblings, tidying up storage tags manually is busywork
  • when you click 'copy traceback' on an error popup, it also copies the software version, your platform, and if you are on a frozen build or running from source
  • the logger now prints version number for every block, just before the timestamp
  • cleaned up a variety of media viewer UI code while working on the scanbar, fixing some misc display bugs
  • moved pixel hash storage responsibility from 'file metadata' to 'similar files' module
  • the similar files system now searches pixel hashes when it is called to do any similar files search. they count as 'exact match' distance
  • when a file gets a new pixel hash, it now sees if any other files have that same hash. if so, it now gets queued up again in the similar files search system, ensuring this match is not missed
  • misc nomenclature cleanup--since we now have both 'pixel hashes' and 'phashes', phashes are now referred to as 'perceptual hashes' everywhere
  • massively refactored the primary table join that drives potential duplicates search. it should work a bit faster now and it is much easier to work with
  • I added pixel dupe and distance search to the standard search results version of the join and the 'system:everything' version, which has several optimisations
  • silenced some shutdown handling in file maintenance that was being printed to log as an error
  • fixed some 'broken object load' error handling to print the timestamp of the specific bad object, not whatever timestamp was requested. this error handling now also prints the full dump name and version to the log, and version to the exported filename. I was working with a user who had broken subs this week, and lacking this full info made things just a little trickier to put back together
  • fixed some drag and drop handling where it was possible to drop thumbnails on a certain location of a page of pages that held an empty page of pages but it would not create a new child media page to hold them
  • misc serverside db code cleanup
  • fixed python 3.10 type bugs in window coordinate saving and Qt image generation from buffer (issue #1027)

• version 465

  • misc

  • fixed a recent bug in wildcard search where 't\*g' unnamespaced wildcards were not returning namespace results
  • sped up multi-predicate wildcard searches in some unusual file domains
  • the file maintenance manager is now more polite about how it works. no matter its speed settings, it now pauses regularly to check on and wait until the main UI is feeling good to start new work. this should relieve heavier clients on older machines who will get a bunch of new work to do this week
  • 'all local files' in _review services_ now states how many files are awaiting physical deletion in the new deferred delete queue. this live updates when the values change but should be 0 pretty much all the time
  • 'all known files' in _review services_ also gets a second safety yes/no dialog on its clear deleted files record button
  • updated the gelbooru 0.2.x gallery page parser, which at some point had been pulling 'delete from favourites' links when running the login-required 'gelbooru favorites by user id' downloader!!! it was deleting favourites, which I presume and hope was a recent change in gelbooru's html. in any case, the parser now skips over any deletion url (issue #1023)
  • fixed a bad index to label conversion in a common database progress method. it was commonly saying 22/21 progress instead of 21/21
  • fixed an error when manage tags dialog posts tags from the autocomplete during dialog shutdown
  • fixed a layout issue with the new presentation import options where the dropdowns could grow a little tall and make a sub-panel scrollbar
  • added handling for an error raised on loading an image with what seems to be a borked ICC profile
  • increased the default per-db-file cache size from 200MB up to 256MB

  • some new options

  • the default tag service in the manage tags dialog (and some similar 'tag services in a row of tabs' dialogs) is reset this week. from now on, the last used service is remembered for the next time the dialog is opened. let's see how that works out. if you don't like it, you can go back to the old fixed default setting under the 'tags' options panel
  • added a checkbox to the 'search' options panel that controls whether new search pages are in 'searching immediately' or 'searching paused' state (issue #761)
  • moved default tag sort from 'tags' options panel to 'sort/collect'

  • deleted files and ipfs searchability

  • wrote a new virtual file service to hold all previously deleted files of all real file services. this provides a mapping cache and tag lookup cache allowing for fast search of any deleted file domain in the future
  • ipfs services also now have mapping caches and tag search caches
  • ipfs services are now searchable in the main file search view! just select them from the autocomplete dropdown file domain button. they have tag counts and everything
  • it will take some time to populate the new ipfs and deleted files caches. if you don't have much deleted files history and no ipfs, it will be a few seconds. if you have a massive client with many deleted/ipfs files and many tags, it could be twenty minutes or more

  • 'has icc profile' now cached in database

  • the client database now keeps track of which image files have an icc profile. this data is added on file import
  • a new file maintenance task can generate it retroactively, and if a file is discovered to have an icc profile, it will be scheduled for a thumbnail regeneration too
  • a new system predicate, 'system:has icc profile', can now search this data. this system pred is weird, so I expect in future it will get tucked into an umbrella system pred for advanced/rare stuff
  • on update, all your existing image files are scheduled for the maintenance task. your 'has icc profile' will populate over time, and thumbnails will correct themselves

  • pixel hash now cached in database

  • the client database now keeps track of image 'pixel hashes', which are fast unique identifiers that aggregate all that image's pixels. if two images have the same pixel hash, they are pixel duplicates. this data is added on file import
  • a new file maintenance task can generate it retroactively
  • on update, all your existing image files are scheduled for the maintenance task. it'll work lightly in the background in prep for future duplicate file system improvements

  • boring search code cleanup

  • fixed a bug where the tag search cache could lose sibling/parent-chained values when their count was zeroed, even though they should always exist in a domain's lookup
  • fixed up some repository reset code that was regenerating surplus tag search data
  • with the new deleted files domain, simplified the new complex domain search pattern
  • converted basic tag search code to support multiple location domains
  • cleaned up some search cache and general table creation code to handle legacy orphan tables without error
  • misc tag and mapping cache code and combined local files code refactoring and cleanup

• version 464

  • image icc

  • images with embedded icc colour metadata are now normalised (to sRGB) like the rest of media rendering in hydrus. ICC can often mean photos, where a nice camera will apply ICC data to compensate for camera defects or general lighting information, or it can mean normal digital images where the software attached extra colour data when it was saved
  • the image will now be rendered with 'fixed' colours in the media viewer, and new thumbnails should be good too. it applies early in image load and should work in all cases hereon, on both client and server
  • images with an ICC will take a little longer to initially load. I'd estimate 10-50ms extra for most. one user with many ICC images discovered 10% of their collection had an ICC. I don't think the delay will be terrible IRL, but see how you get on and let me know! maybe giganto patreon pngs will have a fresh surprise for us
  • future expansions here will be a database cache of ICC images and system:has icc, perhaps a button to click the ICC application on and off live in the media viewer, and then maybe options to load up and switch an ICC for your display

  • better physical file delete

  • both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
  • the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
  • on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
  • when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
  • some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
  • thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file

  • misc

  • fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
  • fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
  • fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)

  • boring image stuff

  • wrote some methods to check for and pull ICC profile bytes from an image with PIL
  • wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
  • cleaned up some PIL/OpenCV image load and normalisation code
  • the decompression bomb check is now quicker for images with rotation
  • dequantization is now applied to PIL on all image load by default, it doesn't have to be invoked separately
  • some metadata parsing like 'get duration of gif frames' is now faster for images not in RGB or RGBA color

  • boring delete code cleanup

  • wrote a heap of new 'is an orphan' filtering logic for client and server
  • wrote a daemon job for physical file deletion and plugged it into a new database queue for pending deferred file deletes
  • client physical file delete now works off the normal lightweight job scheduler, previously it had its own mainloop thread
  • optimised complex file domain file filtering a little
  • the 'clear orphan files' job in the client now uses the same updated orphan logic as the new physical delete code. it now won't clear out files in upload limbo
  • fixed an issue with re-storing a file in a server after one of its file repositories had previously deleted it. this never mattered previously, when files were never physically deleted, but now the code is brushed up to work properly
  • cleaned up some server db code, including the read command method lookup
  • moved client 'hash exists?' test down to the master definitions module

• version 463

  • misc

  • ogv files (ogg with a video stream) are now recognised by the client! they will get resolution, duration, num frames and now show in the media viewer correctly as resizeable videos. all your existing ogg files will be scheduled for a rescan on update
  • wrote new downloader objects to fix deviant art tag search. all clients will automatically update and should with luck just be working again with the same old 'deviant art tag search' downloader
  • added prototype copy/paste buttons to the manage ratings dialog. the copy button also grabs 'null' ratings, let me know how you get on here and we'll tweak as needed
  • file searches with namespaced and unnamespaced tags should now run just a little faster
  • most file searches with multiple search predicates that include normal tags should now run just a little faster
  • the file log right-click menu now shows 'delete x yyy files from the queue' for deleted, ignored, failed, and skipped states separately
  • the tag siblings + parents display sync manager now forces more wait time before it does work. it now waits for the database and gui to be free of pending or current background work. this _should_ stop slower clients getting into hangs when the afterwork updates pile up and overwhelm the main work
  • the option 'warn at this many pages' under _options->gui pages_ now has a max value of 65535, up from 500. if you are a madman or you have very page-spammy subscriptions, feel free to try boosting this super high. be warned this may lead to resource limit crashes
  • the 'max pages' value that triggers a full yes/no dialog on page open is now set as the maximum value of 500 and 2 x the 'warn at this many pages' value
  • the 'max pages' dialog trigger now only fires if there are no dialogs currently open (this should fix a nested dialog crash when page-publishing subscriptions goes bananas)
  • improved error reporting for unsolveable cloudflare captcha errors
  • added clarification text to the edit subscription query dialog regarding the tag import options there
  • added/updated a bunch of file import options tooltips

  • new presentation import options

  • the 'presentation' section of 'file import options' has been completely overhauled. it can do more, and is more human-friendly
  • rather than the old three checkboxes of new/already-in-inbox/already-in-archive, you now choose from three dropdowns--do you want all/new/none, do you want all/only-inbox/inbox-too, and do you want to my-files/and-trash-too. it is now possible to show 'in inbox' exclusively, at the time of publish (e.g. when you highlight)
  • added a little help UI text around the places presentation is used
  • the downloader and watcher page's list right-click menu entries for 'show xxx files' is now a submenu, uses the new presentation import options, shows 'show inbox files', and if you click on one row it says what the default is and excludes other entries if they are duplicates

  • boring presentation import options stuff

  • presentation options are now in their own object and will be easier to update in future
  • the 'should I present' code is radically cleaned up and refactored to a single central object
  • presentation filtering in general has more sophisticated logic and is faster when used on a list (e.g. when you highlight a decent sized downloader and it has to figure out which thumbs to show). it is now careful about only checking for inbox status on outstanding files
  • presentation now always checks file domain, whereas before this was ad-hoc and scattered around (and in some buggy cases lead to long-deleted files appearing in import views)
  • added a full suite of unit tests to ensure the presentation import options object is making the right decisions and filtering efficiently at each stage

  • boring multiple local file services work

  • I basically moved a bunch of file search code from 1 file services to n file services

  • the file storage module can now filter file ids to a complex location search context
  • namespace:anything searches of various sorts now use complex location search contexts
  • wildcard tag searches now use complex location search contexts
  • counted tag searches now use complex location search contexts
  • search code that uses complex location search contexts now cross-references its file results in all cases
  • I now have a great plan to add deleted files search and keep it working quick. this will be the next step, and then I can do efficient complex-location regular tag search and hopefully just switch over the autocomplete control to allow deleted files search

• version 462

  • misc

  • fixed a recent serious regression that could cause a crash when playing audio in mpv (issue #1007)
  • the main importer file log now does 'get next/all/count imports with status y' calls significantly faster, particularly on very large lists. these calls happen all the time for different status text changes and general 'which import to try next?' work. all busy downloader situations should see CPU gains to regular and background work
  • fixed a problem where importing with the min/max file resolution options set would give a typo error when the size was violated rather than 'ignored'
  • I think I have fixed an issue with subscriptions not wanting to run a query if by random accident that query has an invalid URL selected as the query's 'example url' for various pre-work login and bandwidth tests
  • hydrus can now capture duration/fps of videos that specify two very close fps, e.g. 60 and 59.99. previously, these would default to the 24 fallback and could cause some weirdness in mpv
  • replaced the default pixiv artist page api parser with one that fetches the newer url format, matching the tag search. existing users will see no automatic change but will receive the new parser, so if you are a big pixiv user, you might like to switch 'pixiv artist gallery page api' to the 'new urls' parser variant under _network->downloader components->manage url class links_. note that if you do this, your pixiv artist subscriptions will do a mini-resync (which involves some wasted time/bandwidth) as their urls change!

  • network redirect improvements

  • gallery page jobs now give their child 'next gallery page' jobs themselves as a referrer
  • when the gallery downloader gets a 3XX redirect, the file import objects and next gallery pages it makes now get the redirected URL as referral url (previously, it used the original gallery url)
  • when the post downloader gets a 3XX redirect, the redirected url is now added as a primary source url
  • when the post downloader gets a 3XX redirect, child import objects and direct file downloads now get the redirect URL as referral url (previously, it used the original post url)
  • when the raw file downloader gets a 3XX redirect, the redirected url is now added as a primary source url
  • when the raw file downloader gets a 3XX redirect to a Post URL, it now tries to queue that URL back up in the file log, just like when a gallery fetch comes back with a Post URL. some safety code stops potential loops here as well

  • new services

  • a new client now starts with a second local tag service called 'downloader tags'. default tag import options are now initialised in a fresh client to pull all file post tags to this service. this should relieve new users' confusion about setting up default tag import options
  • similarly, a new client now starts with a like/dislike rating service called 'favourites'. existing users who have no rating services will be asked if they want to get it on update. many users are unaware of the rating system, so this is a chance to play with it
  • the 'getting started with downloading' 'and '...with ratings' help has some updated screenshots and talk about the new default services and parsing

  • database fixes

  • fixed a very slow database query that was involved with file search using unnamespaced tags when other search predicates had already limited the scope of search
  • fixed a similar slow query when the 'bad' search predicate was 'namespace:anything', particularly when the namespace is a wildcard
  • fixed the 'clear orphan tables' database maintenance routine. it had decayed due to bit rot and was deleting new repository processing tracking tables. the routine is now plugged directly into the new database modules system, and any module now can be upgraded to discover surplus service tables. the system has been updated to permit the detection and removal of duplicate tables in the wrong database file, and it also makes a statement if no orphan tables were found
  • the 'metadata resync' repository maintenance task now removes surplus file rows from the processing tracking tables
  • the 'metadata resync' repository maintenance task now corrects content type rows in the main processing tracking table
  • the process of registering updates to a repository is now a little faster, more reliable, repairs basic damage, and keeps more good work when damage needs to be repaired
  • I _think_ the users who were still getting PTR processing errors related to a database confusion about content/definitions update files should now be fixed after another full metadata resync! please let me know if there are still problems

• version 461

  • misc

  • the text on the animation scanbar is now center-vertically aligned and should look better on taller and thinner scanbars (issue #998)
  • the scanbar now reports better frame number and current time for the mpv player when the current video is very short or has very few frames. screamer gifs should now report 2/2 frames if you scan to the right, not like 97/2
  • fixed using the mpv player with an embed button (it previously was staying hidden even after embed was clicked) (issue #999)
  • the 'search' submenu when you right-click on tags in certain locations now shows add/exclude namespace:anything if all the selected tags share the same namespace
  • as an experiment that I think will be bulked out into proper shortcuts later, and maybe actual +/- buttons like you'll see on a booru, if you activate the 'selection tags' listbox (double-click or enter key) while ctrl is down, it now excludes the selected tags from the current query
  • when you paste query texts into the edit subscription dialog, those queries already in the sub _and_ DEAD will now be revived (it does a 'check now' on them). the dialog reports this
  • when editing subscriptions, the way it waits for the current subs to stop running is improved. it is now separate from the global 'pause subs' variable, so big delays here (e.g. waiting a long time to open the dialog, then hitting 'pause' on the network menu, which was secretly a logically messy unpause) should be less able to run into trouble
  • watchers now sort DEAD and 404 separately when sorting the status column (previously they were sorted by their now-defunct 'next check time')
  • I think I improved the speed of the new subscriptions guaranteed shutdown. I think I also fixed a shutdown hang on some lagging async jobs. there are a couple of reports of hanging shutdown, so let me know if this changes
  • I moved the autocomplete options from 'gui pages' to 'search', and I brushed up the layout and tooltips there generally

  • file parsing

  • clip files with canvas size units in mm, cm, inches, or points are now parsed correctly! thanks to the user who helped here! turns out a point is 1/72 (two grossths :^)) of an inch
  • clip animations now get the number of frames and duration of the first timeline!
  • all clip files will reparse for fixed resolution and duration and make new thumbs as needed on update
  • hydrus file parsing should now detect the duration of video and audio with 10 or more hours duration
  • hydrus now gets a more accurate duration estimate for files with bonkers duration/start_offset pairs, for instance "Duration: 127:57:31.25, start: 460633.291000". if you ever saw a 7MB webm with 5 hour duration (and actually 18 seconds), it could have been this. hydrus now counts frames manually when you get this sort of thing
  • any file with resolution > 360p, a duration over an hour, and size less than 64MB will be scheduled for a file metadata reparse on update

  • client api

  • /get_files/file_metadata now has an optional boolean parameter, 'hide_service_names_tags', default False, which will hide the old 'service_names_to...' tag Objects
  • a unit test tests hide_service_names_tags
  • client api help documentation now talks about hide_service_names_tags
  • client api version is now 22

  • boring code refactoring and cleanup

  • tl;dr: about 60KB total code moved out of client database!
  • moved most combined sibling+parent database code to a new 'tag display' module
  • moved autocomplete counts cache database code to a new 'mappings counts' module, and refactored a whole lot of of misc old a/c table creation and reference code into that module, cleaning things up
  • the 'mappings counts' module is plugged into new repair code and on error repopulates itself as efficiently as regen code currently allows
  • moved tag cache code to a new 'tag search' module, and similar related decoupling refactoring
  • the 'tag search' module is plugged into new repair code and on error repopulates itself very efficiently
  • the 'local tags cache' module can now regenerate itself on boot
  • regenerating the local tags cache now works a little faster and takes less memory
  • I _believe_ client.caches.db can now regenerate all of itself automatically, with no subsequent user actions needed
  • the boot database repair notifications have some quality of life improvements. modules now say whether they think they can recover everything, and there is more guidance on what to do in the different situations
  • during various heavy database work, a common analysis tool now saves a lot of time on regeneration vs generation
  • pubsubs now go through the transaction wrapper, meaning modules can pubsub
  • emergency boot messages (like database trouble) are now printed to the log

• version 460

  • client api

  • /add_tags/add_tags now supports 'service_keys_to_tags' and 'service_keys_to_actions_to_tags'.
  • /add_urls/add_url now supports 'service_keys_to_additional_tags'
  • /get_files/file_metadata now returns with duplicates of the tag structures using 'service_keys_to_statuses_to_(display_)tags'
  • added unit tests for the above
  • updated the client api help for the above
  • I recommend you move from 'service_names_to...' to 'service_keys_to...' when convenient. 'names' was an ugly old hack, and while I am not in a rush to delete it from the client api, I think I will eventually
  • client api version is now 21

  • apngs and clips

  • fixed a problem where the new apng metadata parsing was not completely hooked up, so num_frames was not being parsed correct for the final metadata row even when 'apng' filetype was, leading to some odd '1 frame apng' situations
  • apng parsing now recognises more kinds of apngs--if one has an unusual scaling chunk in the header, this is recognised and the correct animation chunk searched for
  • clip files now get resolution on import and a thumbnail! big thanks to the user who dug up how to extract this--it actually comes from a sqlite file embedded in the clip! (issue #996)
  • on update, all apngs and clips will be scheduled for metadata rescan, and all clips will be scheduled for thumbnail generation

  • misc

  • right-clicking a selection tag and choosing 'select->files with x tag' now obeys the current tag domain (previously, it forced 'all known tags'). so, if you want to quickly select just the files with 'samus aran' on 'my tags', it should be doable
  • the new 'partial' download resumer system will now tolerate two successive empty chunks before throwing a 'this chunk was empty' error. it seems some servers will randomly give an empty chunk at times during 206 transfers
  • cancelling the slideshow custom time dialog no longer raises an error
  • after the build boot problem last week, updated the opencv version in requirements.txts--we are now officially >=4.0.0, <=4.5.3.56. it looks like pyinstaller needs a patch for 4.5.4.58 to work, so we'll wait for that. I am improving my weekly test routine to try to catch this in future
  • also, the windows build no longer includes two copies of an opencv dll. turns out PyInstaller finds this dll ok now (putting it in another location) and it doesn't need to be explicitly added during build
  • added a new help file to the db dir for users who experience crashes as soon as they load videos in mpv when using WASAPI or ASIO drivers. thanks greatly for the user who figured out the mpv.conf solution here (issue #973)

• version 459

  • main highlights

  • to help debugging from screenshots etc..., the client now puts its version name on every window title, like 'review services - hydrus client 459'. (issue #447)
  • the 'main gui title' option is reset and replaced with 'application display name' this week. it now alters the 'hydrus client' part on every window title. the actual 'main gui title' is now "main" lmao
  • wrote a new help document, 'help my media files are broke' in the db directory. this collects the different recovery routines I have developed while helping users after drive failure or other problems cause many missing files or a desynced database and file storage structure. I will be pointing people to this in future, please feel free to do the same
  • two new file maintenance jobs are added: for 'presence' and 'integrity' checks, you can now do 'if has URL, then try to redownload, else remove record'. this tidily combines the two more specific jobs that are commonly run after a hard drive problem. the 'presence' version is now the default selected job and recommended for most simple situations
  • a new easy-select button on the review file maintenance panel lets you select all media files
  • I put some more time into the new duplicate filter zoom locking calculation. thank you to users who sent in examples of my code not working well--I have scaled back what it tried to do. now it will tend to heigh-lock for landscape images and width-lock for everything else _unless_ you are currently viewing the default zoom and that roughly fills the canvas on a dimension and doing the default lock would cause the next image to spill over the screen. the 'solution' here hence targets the 'watermark spilled over' problem more specifically and deals with all combinations of landscape/portrait A/B/canvas better. I'd still like to introduce some zoom locking options here for regular browsing, but pinning down what exactly is useful is trickier than I expected
  • the edit tag import options panel now shows 'THIS CURRENTLY GETS NO TAGS' warning red text if it is non default and no tags are set to parse and there are no additional tags
  • the status bar now shows '1,234 files in 20 collections' when you have just collections or just collections selected (previously, it wrongly said '1,234 collections') (issue #807)
  • macOS clients will now show dialog-created menus in a debug dialog unless the new BUGFIX checkbox under 'gui' options page is unchecked. this _should_ help Big Sur users who are unable to interact with menus created in dialogs like manage tags or manage services. I threw this together, so let me know how this works for you! (issue #986, issue #858)
  • the program now waits specifically for currently running subscriptions to stop work and save themselves before moving on with further shutdown tasks. hand in hand with this, subscriptions are now faster at stopping work on client exit, even when they have no popup message (through which some hackery dackery shutdown signals are sent otherwise) (issue #790)
  • physically deleting thousands of files in one go should no longer lock up the file manager and other systems for ages--physical delete is now serialised and processed on a new threaded mainloop, so it doesn't matter how fast the requests come it, it will chunter at a polite speed and take breaks and should not choke other consumers and freeze up other 'things are great, you can start new work' status checks

  • network job improvements

  • hydrus network jobs now try to resume incomplete responses (previously they just dumped out and tried again from the beginning). if a server provides less content than it said it would, or it explicitly gives us a partial response, we now resume at that point! should fix dowmnloading of longer videos on 8chan.moe
  • hydrus network jobs now send a range header by default, letting servers return 206 (partial content) if they wish
  • SSL errors (cert verification and similar) are now caught in the network engine separately to generic connection errors. they will not be reattempted, and the failure note will display specific error info
  • refactored some response header parsing code, cleaning up how some variables are initialised
  • greatly improved the job reattempt system, resetting variables more neatly
  • improved some response range and content length calculations

  • smaller items, mostly bug fixes

  • fixed a recent typo bug that caused the edit url class dialog to always spawn with 'file url' type set. sorry, this was stupid! (issue #982)
  • the edit url class dialog now sends the 'normalised' url as the example text for the api and referral string converter edit panels
  • fixed the new advanced file deletion 'remember last' checkboxes in _options->files and trash_. they weren't hooked up right, sorry!
  • fixed the tag menu's siblings submenu's copy command where it says 'ideal is "xxx" on: yyy'. despite the correct label, this was sometimes copying a different service's ideal (issue #855)
  • fixed the 'media zooms' text input under _options->media_ not turning off the 'red' invalid mode once its text is again valid
  • when you cancel the 'edit parser' dialog, it shouldn't say 'it looks like you made changes' so much when you didn't make any. the 'has changes' test now ignores some background test data updates that may have happened (issue #875)
  • if a JSON parsing formula is given HTML, the 'cannot parse' error now tries to detect this and present a better error text (issue #888)
  • I _think_ I fixed a problem in the new bytes rendering calculation (where it goes 1018825 to "995KB") where on some unlucky edge-case numbers it could non-determinitively choose different sig figs (e.g. flipping between 994.9KB and 995KB)
  • fixed a couple of file move actions that were unable to move across Windows partitions when the timestamp was before 1980-01-01 (issue #989)
  • mr bones now recognises you are not a new user if you deleted all your files. you can never exit
  • after some testing, it seems like large 'drop table' operations in SQLite sometimes work within seconds but generally take far longer, often working as slow as 10MB/s (and I just talked to a guy for whom it took _days_(!!!). writing a fix to make 'delete service' always run fast for something as large as the PTR will take planning and work, so for now I have attached a warning note to the delete service confirmation dialog
  • updated the file maintenance review panel to newer async code
  • fixed a typo bug in URL export when a file is missing/bad in file maintenance

• version 458

  • quality of life

  • under _options->files and trash_, you can now govern whether the advanced file deletion dialog remembers action and reason. being able to save action (trash, physical delete, vs physical delete and clear history) is new, default off for now, and won't _always_ save (if you are currently set to trash, but the next dialog doesn't have trash as an option, then it won't overwrite to physical delete). if you try it out, let me know how you like it
  • a new option under 'network->pause' now lets you always boot the client with paused network traffic
  • the main file import object now stores primary urls (such as post and file url) separately from source url (which is produced by many parsers and typically refers to another website). a new checkbox in 'file import options' (when in advanced mode) now allows you to not associate primary urls separately to source urls (which is useful in some one-time technical jobs that talk to some unusual proxy etc...)
  • the new import object right-click menu that shows urls now separates primary and source urls, and also shows any referral url
  • when you flip between two images in the dupe filter, the zoom preservation calculation, which previously only locked to the same width, now tries to choose width or height based on the relative ratios of the two images to keep both images completely in view on a canvas zoom start. it should ensure that lower watermark banners stay in view and don't accidentally spill over the bottom of your monitor
  • moved popup toaster options from 'gui' options page to the new 'popup' page
  • added options for whether the popup toaster should update while the client is minimised and while the mouse is on a different monitor than the main gui window. these options now default to false, so if you have any trouble, please try turning them back on
  • a new shortcut action in the 'global' set now flips profile mode on and off. please note for now 'global' only works on main gui and media viewer--I will add a hook to every window in the future!

  • bug fixes

  • you now cannot start an 'upload pending' job for a service more than once at a time. the menu is now disabled for each service while uploading is happening
  • fixed a bug in media load where if the file was not in a specific domain (i.e. somewhere in all known files), its tags would not show implied parents. for non-specific files, this calculation happens on the fly, and previously it was only doing siblings
  • fixed a bug from the somewhat recent file deletion update that meant many files' custom deletion reasons were being overwritten to 'No reason given' when trash was clearing. I am sorry for the inconvenience!
  • fixed an issue with parsing 'string' from html 'script' tags (and perhaps some other 'meta' tag types) on recent versions of the built hydrus release. this should fix the newgrounds gallery parser
  • fixed some gallery parsing error handling, for when the actually fetched url differs from the original and cannot be parsed, and when the actually fetched url redirects straight to a file page from a 1-length search result

  • update file handling bug fixes

  • when repository processing runs into an update file problem, it now specifies if the file was content or definitions type
  • when the database gathers updates to process, it discriminates between definitions and content updates more carefully
  • when a hydrus update file goes through file maintenance and changes filetype, the repository update progress tracker now 'reregisters' that file, updating what content types it can expect from it and clearing out incorrect data
  • during normal update file registration, incorrect old data is now cleared out

  • boring cleanup

  • cleaned some of the positioning code that places icons and text on thumbnails, removing hardcoded numbers and letting custom icons position better
  • cleaned some import url tracking, checking, and association code
  • refactored profile mode and query planner mode switching up out of UI code into the controller
  • added a hefty unit test to test that siblings and parents are transitively applied to mappings correctly for files outside and inside specific file services, and for tag sync and the normal tag pipeline
  • refactored some database file maintenance code to decouple the queue from the worker

• version 457

  • smoother menubar updates

  • improved the way the menubar menus update. rather than generating a whole new (e.g. 'pages') menu and replacing the existing out of date one, now there is a static menu skeleton that has subsections or labels updated in place. this means fewer objects changing, less flicker/jank, and should allow you to upload pending even if you have, say, a bunch of subscriptions running

  • misc

  • thanks to a user's help, the filetype parser now detects pngs (this mostly happens during import) much faster! the problem previously was determining if a png is actually an apng--figuring out if they are truly apngs is now done with very fast file header scanning, rather than the previous method that booted ffmpeg. this brings filetype parse time for pngs down from 50-150ms to 1-2ms
  • getting apng metadata is also now faster. num_frames is now pulled from the file header, it no longer has to be manually counted by ffmpeg
  • clicking the session weight item in the 'pages' menu now gives you more detailed info on your session weight, including on currently closed pages in the undo list
  • stripped out a lot of ancient wx-era safety code that stops the client from doing certain UI work while it is minimised or minimised to tray. also brushed up some ugly update routines for menus refresh and modal message presentation that could lead to a pile-up of updates as soon as the client was unminimised, causing lag or freezes. with luck, the client should be better about restoring itself from minimised to system tray. if you minimise to tray, feedback on how this works out for you would be appreciated
  • when a network job stalls with the 'this domain had some errors recently' message, the cog menu on the widget now allows you to 'scrub domain errors' and try again immediately
  • if your search has system:limit, then any tag search you type in the autocomplete will now search the database, not your thumbnails. previously, the hack to enable this behaviour was to flip 'searching immediately' off. let's see if this new behaviour is ultimately confusing/annoying, I am mixed on it and think this subtle search option needs more thought and UI to make it more obvious and user friendly
  • if you have autocomplete tag search typed, and results from thumbnails displayed, and you flip 'searching immediately' off, the search will now automatically update and give you full database numbers immediately

  • help

  • I moved 'searching with wildcards' from the advanced help to the 'more getting started with files' help here: https://hydrusnetwork.github.io/hydrus/getting_started_more_files.html
  • I also wrote a more detailed description of what the autocomplete dropdown buttons do in that page
  • I also wrote a brief description of how a system:limit query will try to clip according to the current file sort, getting the n 'biggest files' and so on

  • boring code cleanup

  • cleaned some network job widget update calls
  • improved some misc autocomplete search status tracking
  • improved some account object permission checking and tests. accounts now never say they have permissions (e.g. if you click the 'see account permissions' button on review services) if they are banned or expired
  • file and pages menus now uses the new update routine
  • pending menu now uses the new update routine, with an emphasis on anti-jitter so you can interact while it is updating
  • database, network, service, and undo menu now use newer async update code and also use the new update routine
  • cleaned up help and tags menu init code
  • the signal that causes the pending menu to update is now only sent on tag changes if the tag service is a repository (previously, local-only updates were janking this for no reason)
  • the pending menu now updates its sibling/parent numbers when repository processing causes a clever row change to stuff you have pending
  • also, some menubar items that only show when in advanced mode now update their visibility when advanced mode is flipped on or off
  • misc menubar code cleanup and improvements

• version 456

  • misc

  • the client no longer regularly commits a full garbage collection during memory maintenance. this debug-tier operation can take up to 15s on very large clients, resulting in awful lag. various instances of forcing it after big operations complete (e.g. to encourage post-subscription memory cleanup), are now replaced with regular pauses to allow python to clean itself more granularly. this may result in temporary memory bloat for some very subscription-heavy clients, so feedback would be appreciated
  • right-clicking on a single url import item in a 'file log' now shows you all the known hashes, parsed urls, and parsed tags for that item. I hope this will help debug some weird problems!
  • all multi-column lists across the program now convert an enter/return key press into an 'activate' command, as if you had double-clicked. this should make it easier to, for instance, highlight a downloader or shift/ctrl select a bunch of sibling rows and mass-delete (issue #933)
  • the subscription gap filler button now propagates file import options and tag import options from the subscription to the downloader it creates (issue #910)
  • a new 'mpv report mode' now prints a huge amount of mpv debug information to the hydrus log when activated
  • improved how mpv prints log messages to the hydrus log, including immediate log flushing
  • fixed a bug that meant the hydrus server was not saving custom update period or anonymisation period for next boot. thank you for the reports, and sorry for the trouble! (issue #976)
  • cleaned up some database savepoint handling after a serious transaction error occurs
  • the client api now ignores any parameter with a value of null, as if it were not there, rather than moaning about invalid datatypes (issue #922)

  • url classes

  • the edit url class dialog is now broken into two notebook pages--'match rules', which strictly covers how to recognise a url, and 'options', which handles url storage, conversion, and normalisation
  • url classes can now support single-value parameters (a parameter with just a value, not a key/value pair). if turned on, then at least one single-value parameter is required to match the url, and multiple are permitted. a checkbox in the dialog turns this on and a string match lets you determine if the url class matches the received single value params
  • added unit tests to test the new single-value parameter matching
  • fixed an issue where StringMatch buttons were not emitting their valueChanged signal, guess how I discovered that bug this week
  • fixed the insertion of default parameter values when the URL Class has non-alphabetised params
  • refactored and cleaned up some related parsing and string convertion code into new ClientString module

• version 455

  • misc

  • many of the simple system predicates (width, size, duration, etc..) now support the '≠' (not equal) operator! searches only support one of these at once for now (e.g. you can't say height != 640 AND height != 1080--it'll pick one of these pseudorandomly)
  • the watcher page list right-click menu now has 'copy subjects', which copies the selected watchers' 'subject' texts to clipboard
  • the advanced file deletion panel now remembers which reason you last used. it remembers if you selected the 'default' value up top versus specific reason text, and if you enter custom text, it remembers that for next time too
  • the network job widget now shows the current URL as tooltip over the progress gauge, and under the cog menu, where clicking it copies it to clipboard
  • the various menu labels across the program should now show ampersand (&) correctly (e.g. in URLs)
  • the way byte sizes (like 21.7KB or 1.24GB) above 1KB are rendered to strings has been overhauled. they now generally show three significant figures. a new EXPERIMENTAL option in 'gui' options panel lets you change this, but only 2 or 3 are really helpful
  • if a repository clears the message on your account, you no longer get a popup telling you 'hey, new message from server x: ...'
  • the new ≠ system preds should be parseable (but be careful, likely buggy) using the client api's new system predicate parser, with '≠', '!=', 'is not', or 'isn't'
  • cleaned up some old data presentation methods and improved how client specific options are patched in to base hydrus string conversion code

  • ui freezes

  • session pages can now detect if they have had no saveable changes since a certain time. they use this ability to skip redundant session save CPU time for pages with no changes since the last session save
  • for now, since the smallest atom of the session system is a whole page, gallery and watcher pages can only save time if _every_ downloader in the page has had no changes, so in their case this optimisation mostly only applies to completely finished/paused pages. it is still better to have several medium size downloader pages than one gigantic one
  • a new database maintenance task ensures that optimisation cannot accidentally lose a page (from something like an unfortunate timing of a session save after many manual session deletes)
  • the existing optimisation that skips 'last session' save on no changes now initialises its data as the 'last session' is first loaded (rather than on first save), meaning that if there are no changes while the client is open, no new 'last session's will be saved at all
  • misc session save code cleanup

  • database repair, mostly boring

  • a client can now boot with client.caches.db missing and will rebuild that file. almost all of its tables are now able to automatically repopulate (issue #975)
  • all the new modules I have been working on are now responsible for their own repair. this includes missing indices, missing tables, and table repopulation where possible. modules now know which of their tables are critical to a boot, what version each table and index was added, and now manage both initial and later-created service tables and indices
  • essentially, all newer database repair code is now modularised rather than hardcoded. the repair and creation code actually now share the same table and index definitions. the code is more reliable, checkpoints its good work in case of later failure, and will be much easier to maintain and expand in future
  • lots of module repair and initialisation code is refactored and generally given a full pass
  • the core mappings cache regeneration routine now takes transaction checkpoints throughout its job to save progress and reduce journal size
  • master definition critical error detection code is no longer hardcoded!
  • mapping storage repair code is no longer hardcoded!
  • similar files repair code is no longer hardcoded!
  • parent or sibling cache repair repopulation is no longer hardcoded!
  • the local hashes cache module can now repopulate itself during repair
  • the notes fast search table can now repopulate itself during repair
  • the similar files search tree table can now rebuild itself during repair

• version 454

  • misc

  • when a downloader page fetch gives a 500 Server Error, it is now handled better, status numbers are updated to 'failed' quickly, and I believe the post-500 downloader deadlock issue (downloaders staying as 'pending', not 'working', after several of these 500s) should also be fixed (issue #898 maybe, but many other reports also)
  • when finishing a very large archive/delete filter, the UI should not hang so much to commit the changes. the changes may be delayed a second or more, if your client is currently chugging, so if you are a user who hits F5 real quick after committing archive/delete, let me know how you get on. I've tried to mitigate for your situation, but it may not be perfect.(issue #845)
  • the Edit URL Class dialog will now refuse to OK if the API Converter fails.
  • the Edit URL Class dialog will now put up an 'are you sure?' messagebox if the URL Class matches its own example API URL
  • fixed some gallery error handling when a gallery cannot be parsed
  • improved 'cannot parse' error reporting text to include more information
  • the client api /manage_pages/add_files command should now always preserve the sort of both the file_ids or hashes parameter
  • fixed an instance where image bitmaps were being handled incorrectly (issue #876)
  • the client will no longer report shutdown work due for a repository when the work is on a currently paused content type
  • fixed some logic related to the advanced tag option of 'fetch page even if url recognised and already in db'
  • improved the error message when an unlucky duplicate boot causes a locked database error
  • I rearranged and clarified a couple of links in the 'advanced' area of the help index page
  • added some help texts and tooltips to the edit checker options dialog and fixed some borked layout
  • did a quick hack to fix some db repair code that wasn't dealing with some missing module information. I will brush the repair code up in the near future so that modules can repair themselves

  • some text colour stuff

  • the green/red/blue duplicate comparison statements on the duplicate filter's right-hand hover window are now coloured by QSS. a new 'HydrusIndeterminate' object name handles the blue.
  • the green/red background in the advanced OR input is now governed by HydrusValid and HydrusInvalid
  • the various 'help for this panel' blue texts are also now HydrusIndeterminate
  • I have added dark or light HydrusIndeterminate and more Valid/Invalid class definitions to all the default qss styles. they may need some updates, so if you wrote any of these QSS files, please fiddle with them and send me any updates to roll in!
  • added a new user created QSS style called 'Dark Blue', thank you for the submission!
  • all text colours are now dynamically set, either QSS or through (tag colour) options! there are no more hardcoded text colours!

  • media viewer's image tile renderer

  • a new option in 'speed and memory' allows you to change the typical square dimensions of tiles in the tile renderer. default is 768 pixels. you can go bigger to improve accuracy, but it'll cost a little memory and CPU inefficiency
  • the new 'nice' tile size calculator now tests more potential tile sizes, improving precision and reducing stretch-warping along tile borders for unusual zooms
  • the new nice size calculator is now also used when figuring out tile padding, making padding widths that do not cause stretch-warps
  • there remain slight stretch-warps in some bottom-most and right-most tiles
  • a new canvas tile debug report mode now draws blue lines on tile edges in the media viewer and spams some tile number info to popups
  • improved tile coordinate safety checks for extremely small images (e.g. 1x1) when blown up to ridiculous zooms so there are more than one tile per pixel
  • this took a bunch of work, and I am happy that I figured out some solutions, but I believe it may be impossible to get perfect answers here. please try out this new version and let me know how it goes, particularly in the duplicate viewer where warping is obvious. I think ultimately I may replace this with a single tile system that goes over the borders of your screen, eliminating the stitching problems entirely, although this will eat more memory and CPU

• version 453

  • qol and misc

  • the network job status labels around waiting for 'subscription'/'download page'/'watcher' forced wait slots are reworded. now they just say a more plain 'waiting to work' with a time estimate, and if a job does not get a chance to work this check cycle, it says 'a different xxx got the chance to work' for a few seconds.
  • if a network job does not get bandwidth on a check cycle, it now says 'a different network job got the bandwidth' for a few seconds
  • when waiting on bandwidth or gallery work, network jobs should count down more smoothly, one second at a time, not skip a second so often
  • network job widgets are now better about updating the layout of their two text labels. the status text on the left should take all the available pixels much better, sharing with the '64KB/s' speed text as it changes width and disappear
  • added a new user-made darkmode QSS stylesheet called 'OledBlack' to default hydrus, try it out under _options->style_
  • if the tag domain in a search page is other than 'all known tags', the 'selection tags' box, which limits itself to the current domain's tags, now explicitly labels itself with that domain
  • consolidated and optimised the pre-work checks on all importers/downloaders in pages. pages with idling/finished/paused downloaders will consume just a little less CPU and need to talk to fewer important objects
  • renamed the shortcut sets for viewer/preview media windows and clarified that they are mouse only for now. the new seek command works with these, but you'd have to map ctrl+right-click or something
  • improved the system predicate unit tests to catch datatype problems like with last week's hotfix and system:time imported
  • advanced archive/delete stuff: wrote up a neat idea I had about using local parents applied to the PTR to make fast multi-tag processing workflows here: https://hydrusnetwork.github.io/hydrus/advanced_parents.html#parent_favourites

  • bug fixes

  • an important tag search bug is fixed. for some users, files that were imported before a service was added were not appearing in some of that service's search results, or their tag counts were not added in certain tag autocomplete results. this file miscount is fixed, and holes will be filled on database update. it should not take too long to fix, although different users will have different situations
  • this bug was leading to artificially fast PTR processing speeds on some clients as their older files were being skipped. if you have used the client for a long time but only added the PTR recently, sorry if you notice it slow down! it is now working correct!
  • fixed an important bug in the image rendering system that was causing tile artifacts (little lines of double-pixel jank along tile borders) at a variety of regular zoom levels. the way ideal tile size was being calculated was often incorrect, so I have replaced it with a better calculation
  • the system predicate parser can now parse 'system:is not the best quality file of its duplicate group' (only 'isn't' was working, previously) (issue #954)
  • if the collect-by dropdown is fed garbage namespace data from the namespace sort options, it now recovers with a nicer error message (issue #904)
  • misc db code cleanup and minor refactoring

  • client api

  • OR predicates are now supported in the client api! Just nest within the tag list, and it'll bundle the nested list into an OR. there's an example in the client api help
  • some permissions testing in file search is tightened up--now we have OR and system predicates, if you do not submit any regular positive tags, the search permissions have to be 'allow anything'
  • fixed an issue where the client api would let you ask about sha256 hashes of incorrect length (and would ultimately make a master database id for these borked hashes, even the empty string!!). now the client api throws a 400
  • fixed a bug in /manage_pages/get_pages where all pages were marked as 'selected'=true (issue #841)
  • in the client api, if you use missing file_id(s) on a request for a file, thumbnail, metadata about a file, or when trying to add a files to a page, it now gives 404 correctly (rather than 500) (issue #961)
  • added a section to the client api help on variable encoding, including an example of how to convert a python tag list to JSON+URL encoded string
  • added new unit tests for OR pred parsing and the hash length check
  • client api version is now 20

• version 452

  • misc

  • my 'simple' shortcut commands can now store additional variables. to start things off, I have finally added 'seek video' shortcut commands that have back/forwards and second+millisecond values. these should work on the native video viewer and mpv, audio and video. existing users will have to add their own (do it to the 'media viewers - all' set), but new users will get ctrl+left/right for 2.5s back and 5s forwards as the new defaults. let me know if you have any trouble!
  • the maximum number tracked by 'tag as number' system predicate is expanded from -99999999->99999999 to -2^63 -> 2^63 - 1. tag caches will be regenerated on update to store these, it will take a few minutes. the input ui for the system predicate is temporarily limited to -/+2^31, but I'll expand it
  • subscriptions now have a checkbox for 'do not worry about subscription gaps'. if you have a subscription that gets files randomly, or gets an intentionally small sample, this will disable the 'hey, there was a gap, click here to fill it in' popup messages
  • you can now set negative values for the duplicate score weights in options->duplicates
  • also added a weight for the 'nice ratio' duplicate comparison
  • vastly improved the cancel speed for searches in the realm of 'get the files that have any xxx tag', be that all tags or a namespace wildcard, and also some important search setup for various 'all known files' search pages. if you have ever tried to search the PTR raw and run into a three minute uncancellable initial setup lag, it should be gone now!
  • when you right-click on files in a specific local file domain (e.g. trash or my files), the 'view this file's dupes' number check is now run on 'all local files' as well, and if the numbers differ, a second menu is shown for all local files. this should make it easier to chase dupes of trashed files that are still untrashed while also allowing a trash-only search
  • fixed a critical bug in repository mapping processing that was not adding mappings to certain caches for files imported before the repo was added, and/or the new repository 'per content type' processing reset. this mostly manifested as these files not showing up in search results despite the tag being there. there is more work to do here, so it is top priority next week, and likely some maintenance to regen the bad caches

  • boring rewrites for multiple local file services

  • many users have asked for the ability, when multiple local file domains are available, to search multiple file domains at once. I spent time this week doing background work to support this, and a related concept of searching 'deleted' files, which is a current gap in the program and not always covered by 'all known files'. nothing significant changes, but almost all the file search code now works on n file domains rather than 1, but for now n=1 lmao
  • made a new 'database search context' object to handle a virtualised but still simple and fast file 'location view' at the database level
  • the primary file search call is converted to use this object. references to a single file service are replaced with the view or its components
  • all duplicate file search code is moved to the new location search context
  • searching by 'system:import time' now works over multiple domains, although it is a little muddled. in future, import time predicate will have an optional specific file service and do 'import time' vs 'deleted time'
  • 'system:local' and 'system:not local' is adapted so it can still work fast with multiple file domains, sped up worst case 'local' time, and a new optimisation lets it run fast for 'deleted from local files' too
  • sorting search files by import time is now only supported if the search domain is just one domain

• version 451

  • stupid anti-virus thing

  • we have had several more anti-virus false positives just recently. we discovered that at least one testbed used by these companies was testing the 'open html help' checkbox in the installer, which then launched Edge on the testbed, and then launched the Windows Update process for Edge and Skype, which was somehow interacting with UAC and thus considered suspicious activity owned by the hydrus installer process, lmao. thereafter, it seems the installer exe's DNS requests were somehow being cross-connected with the client.exe scan as that was identified as connected with the installer. taking that checkbox out as a test produced a much cleaner scan. there is a limit to how much of this nonsense I will accomodate, but this week we are trying a release without that 'open help' link in the installer, let's see how it goes
  • semi-related, I brushed up the install path message in the installer and clarified help->help will open the help in the first-start welcome popup message

  • misc

  • I fixed a critical bug in tag sibling storage when a 'bad' tag's mapping is removed (e.g. 'delete A->B') and added ('add A->C') in the same transaction, and in a heap of fun other situations besides, that mostly resulted in the newly added sibling being forgotten. the bug was worse when this was on a local tag service via the manage siblings dialog. this problem is likely the cause of some of our weird sibling issues on clients that processed certain repository updates extremely quickly. I will keep investigating here for more issues and trigger another sibling reset for everyone in the future
  • the 'show some random pairs' button on the duplicates page is nicer--the 'did not find any pairs' notification is a popup rather than an annoying error dialog, and when there is nothing found, it also clears the page of thumbs. it also tries to guess if you are at the end of the current search, and if so, it will not do an auto re-fetch and will clear the page without producing the popup message
  • fixed a bug that meant file order was not being saved correctly in sessions! sorry for the trouble!
  • import of videos is now a little faster as the ffmpeg call to check resolution and duration is now retained to check for presence of an audio channel
  • when files are imported, the status messages are now much more granular. large and CPU-heavy files should move noticeably from hash generation to filetype calculation to metadata to actual file copying
  • fixed a database query bug in the new processing progress tracking code that was affecting some (perhaps older) versions of sqlite
  • when you trash/untrash/etc... a file in the media viewer, the top hover text now updates to show the file location change
  • fixed a typo bug in the new content type tracking that broke ipfs pinning yet again, sorry for the trouble! (issue #955)
  • I fleshed out my database pending and num_pending tests significantly. now all uploadable content types are tested, so ipfs should not break at the _db_ level again
  • the page tab menu now clumps the 'close x pages' into a dynamic submenu when there are several options and excludes duplicates (e.g. 'close others' and 'close to the left' when you right-click the rightmost page)
  • the page tab menu also puts the 'move' actions under a submenu
  • the page tab menu now has 'select' submenu for navigating home/left/right/end like the shortcuts
  • fixed some repository content type checking problems: showing petition pages when the user has moderation privileges on a repository, permission check when fetching number of petitions, and permissions check when uploading files
  • fixed a typo in the 'running in wine' html that made the whole document big and bold
  • across the program, a 'year' for most date calculations like 'system:time imported: more than a year ago' is now 365 days (up from 12 x 30-day months). these will likely be calendar calculated correctly in future, but for now we'll just stick with simple but just a bit more accurate
  • fixed a bug in mpv loop-seek when the system lags for a moment just when the user closes the media viewer and the video loops back to start

  • client api

  • expanded my testing system to handle more 'read' database parameter testing, and added some unit tests for the new client api file search code
  • fixed the 'file_sort_asc' in the new client api file search call. it was a stupid untested typo, thank you for the reports (issue #959)
  • fixed 'file_service_name' and 'tag_service_name' when they are GET parameters in the client api
  • I fleshed out the file search sort help to say what ascending/descending means for each file sort type

  • boring database cleanup

  • to cut down on redundant spam, the new query planner profile mode only plans each unique query text once per run of the mode
  • also fixed an issue in the query planner with multiple-row queries with an empty list argument
  • refactored the tag sibling and parent database storage and lookup code out to separate db modules
  • untangled and optimised a couple of sibling/parent lookup chain regeneration calls
  • moved more sibling and parent responsibility to the new modules, clearing some inline hardcoding out of the main class
  • cleaned up a bunch of sibling, parent, and display code generally, and improved communication between these modules, particularly in regards to update interactions and display sync
  • the similar files data tables are migrated to more appropriate locations. previously, they were all in client.caches.db, now the phash definition and file mapping tables are in master, and the similar files search record is now in main

• version 450

  • misc

  • when exporting files from the file export window, a cancellable popup job with progress updates is also created. if you close the window, you can still cancel the job from the popup
  • fixed a crash bug in file export window
  • system:num file relationships (duplicates) now correctly only returns files in the current file search domain (previously, it returned all files, including those previously deleted etc...)
  • I rearranged some of the thumbnail menu file relationships actions menu. I'm not really happy with this, but a shuffle is easier than a full rework
  • fixed the '4k' resolution label replacer, which was looking at 2060 height not 2160 by mistake
  • the phash generation routine (part of the duplicates system, happens on image imports) now uses less memory and CPU for images with an alpha channel (pngs and still gifs), and if those images are taller or wider than 1:30 or 30:1, the phashes are also better quality
  • the 'fill in subscription gap' popup button now correctly boots its created downloader when the action also opens a new downloader page. previously, due to overactive safety code, it would hang on 'pending' until a client restart. related similar 'start downloader after creating page' actions off drag and drop or client api should also be more reliable

  • repositories (also the various improvements in 449-experimental are folded in)

  • fixed an issue with some 'force repository account refresh' code not kicking in immediately
  • when a client sees repository update period change, it now recalculates the metadata next check time
  • fixed a bug with the new repo sync where updates just added from additive sync were not being processed until client restart. related long-term buggy 'do we have this hash in updates?' and 'how many updates are there?' tests for update metadata are also fixed
  • the experimental by-content-type repository reset from last week now leaves pending content in place
  • the reset also now clears cached service info counts for files, tags, and mappings

  • client api

  • the /get_files/search_files command now takes six new parameters for file/tag domain selection and file sort type and order
  • I wrote out some simple help and added some hacky unit tests for these new parameters. it needs another pass for potential bug fixes and readability/specificity (e.g. what does 'asc' for 'sort by ratio' mean?), but let me know how you get on anyway
  • fixed the new system predicate parsing for system:hash with only one hash
  • improved the url system predicate examples in client api documentation
  • client api version is now 19

  • mr bones

  • mr bones now reports the correct numbers for your 'my files' again (and will continue to do so as multiple local file services are added)
  • mr bones now reports total files deleted and their total size
  • mr bones now reports your earliest recorded file import time
  • mr bones now has separate tabs for different stats types. this neatly ditches the giant stack of numbers this was becoming, but I may revisit it. some people who take mr bones screens will prefer all the info in one easy shot, while I others I know would rather the 'viewing habits' stuff were not immediately there. maybe expanding boxes?
  • fixed some mr bones layout

  • boring code cleanup

  • made a new base class for the different database modules to hold cursor and collect common administrative functions
  • all database queries (about 1,200 of them) now go through a single location in the new class
  • a new profile mode, 'query planner' mode, now prints query text and EXPLAIN QUERY PLAN lines to a new profile log. this is a new experimental thing, extremely spammy, but will help with diagnosing very unusually slow queries on individual clients (it'll most likely show up odd sqlite versions, weird data distributions, or un-analysed tables)
  • updated a core function in 'all known files' mappings change autocomplete count adjustment. this seemed to have extremely bad worst case time, and I think it might have been giving some bad counts in unusual situations

• version 449

  • this is an experimental release! please do not use this unless you are an advanced user who has a good backup, syncs with a repository (e.g. the PTR), and would like to help me out. if this is you, I don't need you to do anything too special, just please use the client and repo as normal, downloading and uploading, and let me know if anything messes up during normal operation

  • repository processing split

  • tl;dr: nothing has changed, you don't have to do anything. with luck, your PTR service is going to fix some bad siblings and parents over the next couple of days
  • repositories now track what they have processed on a per-content basis. this gives some extra maintenance tools to, for instance, quickly reset and reprocess your ~150k tag siblings on the PTR without having to clear and reprocess all 1.3 billion mappings too
  • in review services, you now see definition updates and all a repository's content types processing progress independently (just files for a file repo, but mappings, siblings, and parents for a tag repo). most of the time they will all be the same, but each can be paused separately. it is now possible (though not yet super efficient, since definitions still run 100%) to sync with the PTR and only grab siblings and parents by simply pausing mappings in review services
  • I have also split the 'network' and 'processing' sync progress gauges and their buttons into separate boxes for clarity
  • the 'fill in content gaps' maintenance job now lets you choose which content types to do it for
  • also, a new 'reset content' maintenance job lets you choose to delete and reprocess by content type. the nuclear 'complete' reset is now really just for critical situations where definitions or database tables are irrevocably messed up
  • all users have their siblings and parents processing reset this week. next time you have update processing, they'll come back in over about fifteen minutes, and with luck we'll wipe out some years-old legacy bugs and hopefully discover some info about the remaining bugs. most importantly, we can re-trigger this reprocess in just a few seconds to quickly test future fixes
  • a variety of tests such as 'service is mostly caught up' are now careful just to test for the currently unpaused content types
  • if you try to commit some content that is currently processing-paused, the client now says 'hey, sorry this is paused, I won't upload that stuff right now' but still upload everything else that isn't paused. this is a ' service is caught up' issue
  • tag display sync, which does background work to make sure siblings and parents appear as they should, will now not run for a service if any of the services it relies on for siblings or parents is not process synced. when this happens, it is also shown on the tag display sync review panel. this stops big changes like the new sibling/parent reset from causing display sync to do a whole bunch of work before the service is ready and happy with what it has. with luck it will also smooth out new users' first PTR sync too
  • clients now process the sub-updates of a repository update step in the order they were generated on the server, which _may_ fix some non-determinant update bugs we are trying to pin down
  • all update processing tracking is overhauled. all related code and database access techniques have been brushed up and should use less CPU and fail more gracefully

• version 448

  • client api

  • /get_files/search_files now supports most system predicates! simply submit normal system predicate text in your taglist (check the expanded api help for a list of what is supported now) and they should be converted to proper system preds automatically. anything that doesn't parse will give 400 response. this is thanks to a user that submitted a system predicate parser a long time ago and which I did not catch up on until now. with this framework established, in future I will be able to add more predicate types and allow this parsing in normal autocomplete typing (issue #351)
  • this is a complicated system with many possible inputs and outputs! I have tried to convert all the object types over and fill out unit tests, but there are likely some typos or bad error handling for some unusual predicates. let me know what problems you run into, and I'll fix it up!
  • the old system_inbox and system_archive parameters on /get_files/search_files are now obselete. they still work, but I recommend you just use tags now. I'll deprecate them fully in future
  • /get_files/search_files now disables the implicit system limit that most clients apply to all searches (by default, 10,000), so if you ask for a million files, you'll (eventually) get it
  • a new call /manage_pages/add_files now allows you to add files to any media page, just like a file drag and drop
  • in the /get_files/file_metadata call, the tag lists in the different 'statuses' Objects are now human-sorted
  • added a link to https://github.com/floogulinc/hyextract to the client api help. this lets you extract from imported archives and reimport with tags and URLs
  • the client api is now ok if you POST with a utf-8 charset content-type "application/json;charset=utf-8"
  • the client api now tests the types of items within list parameters (e.g. file_ids should be a list of _integers_), raising an appropriate exception if they are incorrect
  • client api version is now 18

  • misc

  • hydrus now supports wave (.wav) audio files! they play in mpv fine too
  • simple psd files now have thumbnails! complicated ones will get a stretched version of the old default psd filetype thumbnail, much like how flash works. all your psd files are queued up for thumbnail regen on update, so they should figure themselves out in the background. this is thanks to ffmpeg, which it turns out can handle simple psds!
  • vacuum returns as a manual operation. there's some new gui under _database->db maintenance->review vacuum data_. it talks about vacuum, shows current free space for each file, gives an estimate of how long vacuum will take, and allows you to launch vacuum on particular files
  • the 'maintenance and processing' option that checks CPU usage for 'system busy' status now lets you choose how many CPU cores must exceed the % value (previously, one core exceeding the value would cause 'busy'). maybe 4 > 25% is more useful than 1 > 50% in some situations?
  • removed the warning when updating from v411-v436. user reports and more study suggest this range was most likely ok in the end!
  • double-clicking the autocomplete tag list, or the current/pending/etc.. buttons, should now restore keyboard focus back to the text input afterwards, in float mode or not
  • the thumbnail 'remote services' menu, if you have file repositories or ipfs services, now appears on the top level, just below 'manage'
  • the file maintenance menu is shuffled up the 'database' menubar menu
  • fixed mr bones! I knew I was going to make a file status typo in 447, and he got it
  • in the downloader system, if a download object has any hashes, it now no longer consults urls for pre-import predictions. this saves a little time looking up urls and ensures that the logically stronger hashes take precedence over urls in all cases (previously, they only took precedence when a non-'looks new' status was found)
  • fixed an ugly bug in manage tag siblings/parents where tags imported from clipboard or .txt were not being cleaned, so all sorts of garbage with capital letters or leading spaces could be entered. all pairs are now cleaned, and anything invalid skipped over
  • the manage tag filter dialog now cleans all imported tag rules when using the 'import' button (issue #768)
  • the manage tag filter dialog now allows you to export the current tag filter with the export button
  • fixed the 'edit json parse rule' dialog layout so if you transition from a short display to a string match that has complicated controls, it should now expand properly to show them all
  • I think I fixed an odd bug where when uploading pending mappings while more mappings were being added, the x/y progress could accurately but unhelpfully continually reset to 0/y, with an ever-decreasing y until it was equal to the value it had at start. y should now always grow
  • hydrus servers now put their server header on a second header 'Hydrus-Server', which should allow them to be properly detectable through a proxy that overrides 'Server'
  • optimised a critical call in the tag mappings update database routine. for a service with many siblings and parents, I estimate repository processing is 2-7% faster
  • optimised the 'add/delete file' database routines in multiple ways, particularly when the file(s) have many deleted tags, and for the local file services, and when the client has multiple tag services
  • brushed up a couple of system predicate texts--things like num_pixels to 'number of pixels'

  • boring database refactoring

  • repository update file tracking and service id normalisation is now pulled out to a new 'repositories' database module
  • file maintenance tracking and database-level file info updates is now pulled out to a new 'files maintenance' database module
  • analyse and vacuum tracking and information generation is now pulled out to a new 'db maintenance' database module
  • moved more commands to the 'similar files' module
  • the 'metadata regeneration' file maintenance job is now a little faster to save back to the database
  • cleared out some defunct/bad database code related to these two modules
  • misc code cleanup, particularly around the stuff I optimised this week

• version 447

  • misc

  • fixed drag and dropping multiple newline separated urls onto the client when those urls come from a generic text source
  • pages now cache their 'ordered' file id list. this speeds up several little jobs, but most importantly should reduce session save time for sessions with tens of thousands of files
  • common file resolutions such as 1920x1080 are now replaced in labels with '1080p' strings as already used in the duplicate system. also added 'vertical' variants of 720p, 1080p, and 4k
  • when a page preview viewer gets a call to clear its current media when it is not currently the page in view, it now recognises that properly. this was happening (a 'sticky' preview) on drag and drops that navigated and terminated on other pages
  • the various 'retry ignored' commands on downloaders now give an interstitial dialog where you can choose to retry 'all', '404s', or 'blacklisted' files only
  • manage tag siblings/parents now disables its import button until its data is loaded. imports that were clicked through before loading were being forgotten due to tangled logic, so for now I'll just disable the button!
  • reduced some more spiky database I/O overhead from the UI's perspective (now savepoints are performed after a result is returned, just like I recently did with transaction commit)
  • duplicate potentials search will now update the y in its x/y progress display if many files have been imported since the search was started and x becomes larger than y (due to y secretly growing)
  • fixed the default 'gelbooru md5' file lookup script. if you have a lookup script with this name, it will be updated to my new default automatically. I don't really like fixing this old system, but I am not sure when I will fit in my big rewrite that will merge it with the normal downloader system, so this is a quick fix for the meantime
  • if you are one of the users who had weird unfixable 404 update file problems with the PTR, please try unpausing and doing a metadata resync one more time this week. fingers crossed, this is fixed. please let me know how you get on too, fixed or not, and also if you have had 'malformed' database problems in the past

  • multi column lists

  • improved the precision of longer text pixel_width->text and text->pixel_width calculations, which are particularly used in the multi-column list state saving system. another multi-column size calculation bug, where lists could grow by 1 character's width on >~60 character width columns on every dialog reopen, is now fixed
  • multi-column lists should now calculate last column width more precisely and accurately regardless of vertical scrollbar presence or recent show/hide
  • the snapping system that locks last column size to 5-character multiples can now snap up or down, increasing error tolerance
  • I added a hack to stop the bug some people had of multi-column lists suddenly growing wide, up to screen width, in a resize loop. I think it works, but as I cannot reproduce this error, please let me know how you get on. resizing the options->external programs panel seems to initiate it reliably for those users affected

  • profile mode

  • all debug profile modes (callto, db, server, menu, pubsub, and ui) are now merged into one mode under help->debug
  • this new mode no longer spams popups, and it only prints 'slow' jobs to the profile log
  • it also makes a new profile log every time it is turned on, using mode start timestamp rather than client boot timestamp, and when profile mode is turned off, there is a popup summary of how many fast and slow jobs passed through during the log time
  • touched up profile code, timing thresholds, summary statements, and the help

  • special update rule this week

  • due to the big file storage rework this week, there's some bit rot in older update routines. 447 cannot update databases older than 411, and it _may_ have trouble updating before 436. if this applies to you, the client will error out or warn you before continuing. I'd like to know what happens to you if you are v411-435 so I can refine these messages

  • boring database refactoring

  • the primary current, deleted, pending, and petitioned files tables are now split according to service, much as I have done with mapping tables in the past. this saves a little space and accelerates many file calculations on large clients. if you have a client database script or patch that inspects 'current_files' or 'deleted_files', you'll now be looking at client_files_x etc.., where x is the service_id, and they obviously no longer have a service_id column
  • a new file storage database module manages these tables, and also some misc file deletion metadata
  • refactored all raw file storage updates, filters, and searches to the new module
  • the mappings and new file storage database modules are now responsible for various 'num files/mappings' metadata calculations
  • most file operations on smaller domains, typically trash or repository update files, will be significantly faster (since the much larger 'my files' table data isn't fattening the relevant indices, and worst case query planning is so much better)
  • cleaned up a ton of file domain filtering code as a result of all this
  • physical file deletion is now much faster when the client has many pending file uploads to a file repository or IPFS service
  • complicated duplicate file operations of many sorts should be a _little_ faster now, particularly on large clients
  • searching files with 'file import time' sort should be a little faster in many situations
  • tag repositories no longer bother going down to the database level to to see if they have any thumbnails to sync with
  • everyone also gets a local file id cache regen this week, it may take a few seconds on update

• version 446

  • misc

  • fixed a typo bug in the latest pending upload routine when it was cancelled/errors out early
  • fixed a problem with the new subscription gap downloader, where when the page opens with the first query, it could sometimes assign 'already in db' to items in that query that were actually 'successful'. some other downloaders may have been rarely hit by this, but it was mostly the gap downloader
  • the client _should_ now support a service host that has path components (e.g. one hosted on a proxy), like myserver.com/hydrus_repo. the port will now be correctly inserted in the address before all requests. hydrus and ipfs both should work, fingers crossed
  • when an admin modifies the account types, the server now only prints the 'updated account type' log record if there were actual changes

  • downloader UI

  • the confusingly named 'file import status' and 'gallery search log' in the downloader system are now renamed 'file log'/'search log'/'check log' for the file import queue, gallery downloader, and watcher respectively. the 'table' bitmap buttons are also replaced with simple easy (and easy to refer to!) label buttons.
  • when you open the file/search/check logs from the downloader page list right-click menu, they now spawn properly inside regular windows, not modal dialogs (which were inhibiting interaction with the rest of the program while open)
  • relabelled the awkward 'even if url/hash recognised' checkboxes in file import options. 'do not skip if' becomes 'force download even if', and the text matches that in tag import options for page content. also improved the tooltip on these checkboxes
  • all of the downloader layout boxes have also been renamed and harmonised with each other. gone are overly technical 'import queue' and 'gallery parsing'. now it is generally 'imports' up top and 'search' or 'checker' etc... below
  • layouts have also been harmonised a little. the url downloader page now has boxes for file vs search URLs, the hard drive import pause button is moved up as other pages have, and several off-by-a-pixel sizer layouts have been fixed

  • null account

  • to further improve server privacy, particularly after the PTR's multiple account switch, all repositories now forget which accounts uploaded which content after a certain age. by default it is 90 days, but you can check in _review services_ once a server updates. this defends against a variety of hypothetical attacks where someone very clever gains access to the raw server database files, maybe years from now, and tries to crawl its anonymous account history for derivable information--now there is no history!
  • it will take some time to retroactively scrub a huge server like the PTR. for the PTR, it is mostly a relative no-op of moving account ids from the old public shared account to a new 'null' account, but it'll still be about 1.2 billion rows! this happens in the background, so the server will still be useable most of the time, but it will have spikes of 'busy' for about one hour every four (i.e. one hour of mostly busy, three hours of free), probably for several days. it may be a pain to try uploading a bunch of stuff in that time, so if you have a million pending mappings, you might like to just give the PTR a break for a few days. once it has fully caught up, the anonymisation should only be 20-60 seconds of 'busy' a day
  • the way the anonymisation works is all serverside services now have a single non-useable 'null account' that will take possession of all content after the delay. the original uploader is lost, and the whole historical record is merged together.
  • the privacy help doc has been updated to talk about the new anonymising system. overall, I think the null account pretty much eliminates the speculative account cross-referencing worries we had, and I am happy

  • admin/janny info

  • for admins, all repositories now have an 'anonymisation period' option that you can edit in the service admin menu, defaulting to 90 days (min 1, max 360). you'll also see summary statements in the server logs as updates are anonymised. anonymisation will kick in two minutes after boot, so if you want to change this value immediately on update, get ready and be quick about it!
  • for jannies who can see accounts, you will see the null account pop up in reference to older content (moreso in future when I expand janny UI and permissions). it being special is highlighted, and various account modifying UI shows it cannot be edited
  • also for jannies/admins, I had to do some wickity woo to get the null account to work without a network update for everyone. if you try to look at the null account on 445 you may get an error. normal users won't run into this, but there's a kind of 'soft' network version update for you today

• version 445

  • misc

  • fixed some weird bugs on the pathname tagging dialog related to removal and re-adding of tags with its 'tags just for selected files' list. previously, in some circumstances, all selected paths could accidentally share the same list of tags, so further edits on a subset selection could affect the entire former selection
  • furthermore, removing a tag from that list when the current path selection has differing tags should now successfully just remove that tag and not accidentally add anything
  • if your client has a pending menu with 'sticky' small tag count that does not seem to clear, the client now tries to recognise a specific miscount cause for this situation and gives you a little popup with instructions on the correct maintenance routine to fix it
  • when pending upload ends, it is now more careful about when it clears the pending count. this is a safety routine, but it not always needed
  • when pending count is recalculated from source, it now uses the older method of counting table rows again. the new 'optimised' count, which works great for current mappings, was working relatively very slow for pending count for large services like the PTR
  • fixed rendering images at >76800% zoom (usually 1x1 pixels in the media viewer), which had broke with the tile renderer
  • improved the serialised png load fix from last week--it now covers more situations
  • added a link, https://github.com/GoAwayNow/Iwara-Hydrus, to Iwara-Hydrus, a userscript to simplify sending Iwara videos to Hydrus Network, to the client api help
  • it should now again be possible to run the client on Windows when the exe is in a network location. it was a build issue related to modern versions of pyinstaller and shiboken2
  • thanks to a user's help, the UPnPc executable discoverer now searches your PATH, and also searches for 'upnpc' executable name as a possible alternative on linux and macOS
  • also thanks to a user, the test script process now exits with code 1 if the test is not OK

  • optimisations

  • when a db job is reading data, if that db job happens to fall on a transaction boundary, the result is now returned before the transaction is committed. this should reduce random job lag when the client is busy
  • greatly reduced the amount of database time it takes to check if a file is 'already in db'. the db lookup here is pretty much always less than a millisecond, but the program double-checks against your actual file store (so it can neatly and silently fill in missing files with regular imports), however on an HDD with a couple million files, this could often be a 20ms request! (some user profiles I saw were 200ms!!! I presume this was high latency drives, and/or NAS storage, that was also very busy at the time). since many download queues will have bursts of a page or more of 'already in db' results (from url or hash lookups), this is why they typically only run 30-50 import items a second these days, and until this week, why this situation was blatting the db so hard. the path existence disk request is pulled out of precious db time, allowing other jobs to do other db work while the importer can wait for disk I/O on its thread. I suspect the key to getting the 20ms down to 8ms will be future granulation of the file store (more than 256 folders when you have more than x files per folder, etc...), which I have plans for. I know this change will de-clunk db access when a lot of importers are working, but we'll see this week if the queues actually process a little faster since they can now do file presence checks in parallel and with luck the OS/disk will order their I/O requests cleverly. it may or may not relieve the UI hangs some people have seen, but if these checks are causing trouble it should expose the next bottleneck
  • optimised a small test that checks if a single tag is in the parent/sibling system, typically before adding tags to a file (and hence sometimes spammed when downloaders were working). there was a now-unneeded safety check in here that I believe was throwing off the query planner in some situations
  • the 'review threads' debug UI now has two new tabs for the job schedulers. I will be working with UI-lag-experiencing users in future to see where the biggest problems are here. I suspect part of it will overhead from downloader thread spam, which I have more plans for
  • all jobs that threads schedule on main UI time are now profiled in 'callto' profile mode

  • site encoding fixes

  • fixed a problem with webpages that report an encoding for which there is no available decoder. This error is now caught properly, and if 'chardet' is available to provide a supported encoding, it now steps in fixes things automatically. for most users, this fixes japanese sites that report their encoding as "Windows-31J", which seems to be a synonym for Shift-JIS. the 'non-failing unicode decode' function here is also now better at not failing, ha ha, and it delivers richer error descriptions when all attempts to decode are non-successful
  • fixed a problem detecting and decoding webpages with no specified encoding (which defaults to windows-1252 and/or ISO-8859-1 in some weird internet standards thing) using chardet
  • if chardet is not available and all else fails, windows-1252 is now attempted as a last resort
  • added chardet presence to help->about. requests needs it atm so you likely definitely have it, but I'll make it specific in requirements.txt and expand info about it in future

  • boring code cleanup

  • refactored the base file import job to its own file
  • client import options are moved to a new submodule, and file, tag, and the future note import options are refactored to their own files
  • wrote a new object to handle current import file status in a better way than the old 'toss a tuple around' method
  • implemented this file import status across most of the import pipeline and cleaned up a heap of import status, hash, mime, and note handling. rarely do downloaders now inspect raw file import status directly--they just ask the import and status object what they think should happen next based on current file import options etc...
  • a url file import's pre-import status urls are now tested main url first, file url second, then associable urls (previously it was pseudorandom)
  • a file import's pre-import status hashes are now tested sha256 first if that is available (previously it was pseudorandom). this probably doesn't matter 99.998% of the time, but maybe hitting 'try again' on a watcher import that failed on a previous boot and also had a dodgy hash parser, it might
  • misc pre-import status prediction logic cleanup, particularly when multiple urls disagree on status and 'exclude previously deleted' is _unchecked_
  • when a hash gives a file pre-import status, the import note now records which hash type it was
  • pulled the 'already in db but doesn't actually exist on disk' pre-import status check out of the db, fixing a long-time ugly file manager call and reducing db lock load significantly
  • updated a host of hacky file import unit tests to less hacky versions with the new status object
  • all scheduled jobs now print better information about themselves in debug code

• version 444

  • gave the 'access keys' and 'privacy' help pages a complete pass. the access keys section talks about the read-only shared key, and how to generate you own account, and the privacy section now compiles, as comprehensively as I could, our various discussions about multiple accounts, what you shouldn't upload to the PTR (basically your own name lol), self-signed https certificates, and what information is actually stored on an account
  • expanded the 'getting started - installing' help page with a 'how to run the client' section, including bundling the excellent Linux virtual memory guide written by a user
  • fixed the new 'fill in subscription with gap downloader' button, which was initialising with the wrong downloader at times (usually on the first gap downloader opened, when it opened a new page with it)
  • you can now set 'all known files' for the tag autocomplete in 'write' contexts (e.g. manage tags dialog) when not in advanced mode
  • cleaned up how a variety of delayed UI calls are registered and present information about themselves. every UI job now has a nice human name for debug purposes. this should improve program stability and clear some odd rare errors when closing some dialogs (this mostly affected certain linux users)
  • when an asynchronous UI job fails with a dead window, or if fails to publish to its window for a non-dead reason and then the window dies before that failure returns, the error handling code now catches and silences the error. an example of this would be clicking 'refresh account' on review services, then closing the window before the lagging job raises 'connection failure'
  • when windows are rescued from off screen, their frame key is now stated in the popup note
  • if your version of OpenCV is unable to load PNG files, your client should now be able to load serialised object PNGs (like those in the downloader system) correctly (the same PIL fallback for regular media files now works for deserialisation too)
  • the hydrus log path is finally month-zero-padded, ha ha ha
  • misc cleanup and label fixes

• version 443

  • quality of life

  • when subscriptions hit their 'periodic file limit', which has always been an overly technical term, the popup message now explains the situation in better language. it also now provides a button to automatically fill in the gap via a new gallery downloader page called 'subscription gap downloaders' that gets the query with a file limit five times the size of the sub's periodic download limit
  • I rewrote the logic behind the 'small initial sync, larger periodic sync' detection in subscription sync, improving url counting and reliability through the third, fourth, fifth etc... sync, and then generalised the test to also work without fixed file limits and for large-gallery sites like pixiv, and any site that has URLs that often produce multiple files per URL. essentially, subs now have a nice test for appropriate times to stop url-adding part way through a page (typically, a sub will otherwise always add everything up to the end of a page, in order to catch late-tagged files that have appeared out of order, but if this is done too eagerly, some types of subs perform inefficiently)
  • this matters for PTR accounts: if your repository account does not have permissions to upload something you have pending, the popup message talking about this now hangs around for longer (120 seconds), explains the issue better, and has a button that will take you directly to the _manage services_ panel for the service and will hit up 'check for auto-account creation'
  • in _manage services_, whenever you change the credentials (host, port, or access key) on a restricted service, that service now resets its account to unknown and flags for a swift account re-fetch. this should solve some annoying 'sorry, please hit refresh account in _review services_ to fix that manually' problems
  • a new option in maintenance and processing allows you to disable idle mode if the client api has had a request in the past x minutes. it defaults disabled
  • an important improvement to the main JobScheduler object, which farms out a variety of small fast jobs, now massively reduces Add-Job latency when the queue is very busy. when you have a bunch of downloaders working in the background, the UI should have much less lag now
  • the _options->speed and memory_ page has a full pass. the thumbnail, image, and image tile caches now have their own sections, there is some more help text, and the new but previously hardcoded 10%/25% cache and prefetch limits are now settable and have dynamic guidance text that says 'about a 7,245x4,075 image' as image cache options change
  • all the cache options on this page now apply instantly on dialog ok. no more client restart required!

  • other stuff, mostly specific niche work

  • last week's v441->442 update now has a pre-run check for free disk space. users with large sessions may need 10GB or more of free space to do the conversion, and this was not being checked. I will now try to integrate similar checks into all future large updates
  • fixed last week's yandere post parser link update--the post url class should move from legacy moebooru to the new yandere parser correctly
  • the big maintenance tasks of duplicate file potentials search and repository processing will now take longer breaks if the database is busy or their work is otherwise taking a long time. if the client is cluttered with work, they shouldn't accidentally lag out other areas of the program so much
  • label update on ipfs service management panel: the server now reports 'nocopy is available' rather than 'nocopy is enabled'
  • label update on shortcut: 'open a new page: search page' is now '...: choose a page'
  • fixed the little info message dialog when clicking on the page weight label menu item on the 'pages' menu
  • 'database is complicated' menu label is updated to 'database is stored in multiple locations'
  • _options->gui pages->controls_ now has a little explanatory text about autocomplete dropdowns and some tooltips
  • migrate database dialog has some red warning text up top and a small layout and label text pass. the 'portable?' is now 'beneath db?'
  • the repository hash_id and tag_id normalisation routines have two improvements: the error now shows specific service_ids that failed to lookup, and the mass-service_hash_id lookup now handles the situation where a hash_id is mapped by more than one service_id
  • repository definition reprocessing now corrects bad service_id rows, which will better heal clients that previously processed bad data
  • the client api and server in general should be better about giving 404s on certain sorts of missing files (it could dump out with 500 in some cases before)
  • it isn't perfect by any means, but the autocomplete dropdown should be a _little_ better about hiding itself in float mode if the parent text input box is scrolled off screen
  • reduced some lag in image neighbour precache when the client is very busy

  • boring code cleanup

  • removed old job status 'begin' handling, as it was never really used. jobs now start at creation
  • job titles, tracebacks, and network jobs are now get/set in a nicer way
  • jobs can now store arbitrary labelled callable commands, which in a popup message becomes a labelled button
  • added some user callable button tests to the 'make some popups' debug job
  • file import queues now have the ability to discern 'master' Post URLs from those that were created in multi-file parsing
  • wrote the behind the scenes guts to create a new downloader page programmatically and start a subscription 'gap' query download
  • cleaned up how different timestamps are tracked in the main controller

• version 442

  • gui sessions

  • gui sessions are no longer a monolithic object! now, each page is stored in the database separately, and when a session saves, only those pages that have had changes since the last save are written to db. this will massively reduce long-term HDD writes for clients with large sessions and generally reduce lag during session save intervals
  • the new gui sessions are resilient against database damage--if a page fails to load, or is missing from the new store, its information will be recorded and saved, but the rest of the session will load
  • the new page storage can now be shared across sessions. multiple backups of a session that use the same page now point to the same record, which massively reduces the size of client.db for large-sessioned clients
  • your existing sessions and their backups will obviously be converted to the new system on update. if any fail to load or convert, a backup of the original object will be written to your database directory. the conversion shouldn't take more than a minute or two
  • the old max-object limit at which a session would fail to save was around 10M files and/or 500k urls total. it equated to a saved object of larger than 1Gb, which hit an internal SQLite limit. sessions overall now have no storage limit, but individual pages now inherit the old limit. Please do not hurry to try to test this out with giganto pages. if you want to make do a heap of large long-term downloaders, please spread the job across several pages
  • it seems URLs were the real killer here, so I am rebalancing it so URLs now count for 20 weight each. the weight limit at which point a _page_ will now fail to save, and the client will start generally moaning at you for the whole session (which can be turned off in the options), is therefore raised to 10M. most of the checks are still session-wide for now, but I will do more work here in future
  • if you are in advanced mode, then each page now gives its weight (including combined weight for 'page of pages') from its tab right-click menu. with the new URL weight, let's get a new sense of where the memory is actually hanging around IRL
  • the page and session objects are now more healthily plugged into my serialisation system, so it should be much easier to update them in future (e.g. adding memory for tag sort or current file selection)

  • the rest

  • when subscriptions die, the little reporting popup now includes the death file velocity ('it found fewer than 1 files in the last 90 days' etc...)
  • the client no longer does vacuums automatically in idle time, and the soft/full maintenance action is removed. as average database size has grown, this old maintenance function has increasingly proved more trouble than it is worth. it will return in future as a per-file thing, with better information to the user on past vacuums and empty pages and estimates on duration to completion, and perhaps some database interrupt tech so it can be cancelled. if you really want to do a vacuum for now, do it outside the program through a SQLite intepreter on the files separately
  • thanks to a user submission, a yande.re post parser is added that should grab tags correct if you are logged in. the existing moebooru post parser default has its yande.re example url removed, so the url_class-parser link should move over on update
  • for file repositories, the client will not try to sync thumbnails until the repository store counts as 'caught up' (on a busy repo, it was trying to pull thumbs that had been deleted 'in the future'). furthermore, a 404 error due a thumb being pulled out of sync will no longer print a load of error info to the log. more work will be needed here in future
  • I fixed another stupid IPFS pin-commit bug, sorry for the trouble! (issue #894)
  • some maintenance-triggered file delete actions are now better about saving a good attached file delition reason
  • when the file maintenance manager does a popup with a lot of thumbnail or file integrity checks, the 'num thumbs regenned/files missing or invalid' number is now preserved through the batches of 256 jobs
  • thoroughly tested and brushed up the 'check for missing/invalid files' maintenance code, particularly in relation to its automatic triggering after a repository processing problem, but I still could not figure out specifically why it is not working for some users. we will have to investigate and try some more things
  • fixed a typo in client api help regarding the 'service_names_to_statuses_to_display_tags' variable name (I had 'displayed' before, which is incorrect)

  • build fixes

  • fixed the new Linux and Windows extract builds being tucked into a little 'ubuntu'/'windows' subfolder, sorry for the trouble! They should both now have the same (note Caps) 'Hydrus Network' as their first directory
  • fixed the new Linux build having borked permissions on the executables, sorry for the trouble!
  • since I fixed the urllib3 problem we had with serialised sessions and Retry objects, I removed it from the requirements.txts. now 'requests' can pull what it likes
  • after testing it with the new build, it looks like I was mistaken years ago that anyone could run hydrus from source when inside a 'built' release (due to dll conflicts in CWD vs your python install). maybe this is now only true in py3 where dll loading is a little different, but it was likely always true and my old tests only ever worked because I was in the same/so-similar environment so the dlls were not conflicting. in any case the builds no longer include the .py/.pyw files and the 'hydrus' source folder, since it just doesn't seem to work. if you want to run from source, grab the actual source release in a fresh, non-conflicting directory. I've updated the help regarding this, sorry for any trouble or confusion you have ever run into here
  • updated the running from source document to talk more about actually getting the source and fleshed out the info about running the scripts

  • misc boring refactoring and db updates

  • created a new 'pages' gui module and moved Pages, Thumbs, Sort/Collect widgets, Management panel, and the new split Session code into it
  • wrote new container objects for sessions, notebook pages, and media pages, and wrote a new hash-based data object for a media page's management info and file list
  • added a table to the database for storing serialised objects by their hash, and updated the load/save code to work with the new session objects and manage shared page data in the hashed storage
  • a new maintenance routine checks which hashed serialisables are still needed by master containers and deletes the orphans. it can be manually fired from the _database->maintenance_ menu. this routine otherwise runs just after boot and then every 24 hours or every 512MB of new hashed serialisables added, whichever comes first
  • management controllers now discard the random per-session 'page key' from their serialised key lookup, meaning they serialise the same across sessions (making the above hash-page stuff work better!)
  • improved a bunch of access and error code around serialised object load/save
  • improved a heap of session code all over
  • improved serialised object hashing code

• version 441

  • misc

  • after successful testing, all the master builds are now made on github rather than my home dev situation. the clients now work off python 3.8, and several security libraries (e.g. OpenSSL) are now always going to be latest, so there should be several quiet performance and reliability improvements across the program. there are no special install instructions--normal update seems to go fine--but let me know if you do have any trouble. big thanks to the user who did the leg work on developing the workflow build scripts here
  • if you are in advanced mode, namespace file sorting now allows you to set the 'tag display context' on which it will sort. this appears as a new menu button or a button list selection dialog wherever you edit namespace file sorts. if you are not in advanced mode, the default is the 'display tags' I switched to last week (i.e. before any tags are hidden by your tag display options)
  • namespace sort has some related code cleanup. the 'defaults' object is updated and moved to the newer options object
  • the new tiled renderer now checks for rounding errors in zoom calc, which in some cases was giving a single extra (non-existing) native pixel row or column on rightmost or bottommost tile samples
  • the new tiled renderer now double-checks clip regions for validity before attempting to crop
  • improved the reported error information when a tile fails to render
  • when pasting an uneven number of tags into manage siblings/parents, the error is now a nicer popup dialog. I'm pursuing a related error here--if you get this a bunch, please let me know what more info you discover
  • when repositories fail to fetch the update hashes to process, they now force a metadata resync. any processing error should force a metadata resync now
  • added a default url class for the new pixiv _artist_ page format
  • fixed a recent typo bug with ipfs pinning

  • client api additions

  • the client api has a new /manage_headers/set_user_agent call, which is a simple hack for now for external programs to set the 'Global' User-Agent. it should allow for some CloudFlare solutions when just copying cookies is not enough
  • the client api has a new /get_services call, which talks about more services and also exposes service_keys for the first time, which are likely to be useful in future. check out the help for an example. the old /add_tags/get_tag_services call is now deprecated, please move to the new call
  • the client api /version call now responds with 'hydrus_version' as well, which this week will be 441
  • the client api now has a semi-experimental /manage_database/lock system, just like the server's. a new 'manage database' permission is added for this. don't play around with this system idly.
  • the client api should now support sha256 hash parameters if they start with a type prefix like 'sha256:0123789abcdef...'
  • the client and server's database lock commands now wait up to five seconds for the database to finish disconnecting to respond
  • expanded client api unit tests to cover the above
  • the client api version is now 17

  • boring multiple local file services work

  • the main search object now stores the file domain using a new 'location context' object that will in future hold multiple file services and can say whether we should search files currently in a domain, or those once deleted from it. a variety of back-end search code has been updated to deal with this more flexible situation
  • removed more static references to the single 'my files' domain in db and related code. in a couple places, like mr. bones, it now fetches 'all local files', but this will likely be updated in future to a new umbrella 'all non-trash, non-repo-update-files local files' service

• version 440

  • tiled renderer

  • the tiled renderer now has an additional error catching layer for tile rendering and coordinate calculation and _should_ be immune to to the crashes we have seen from unhandled errors inside Qt paint events
  • when a tile fails to render, a full black square will be used instead. additional error information is quickly printed to the log
  • fixed a tile coordinate bug related to viewer initialisation and shutdown. when the coordinate space is currently bugnuts, now nothing is drawn
  • if the image renderer encounters a file that appears to have a different resolution to that stored in the db, it now gives you a popup and automatically schedules a metadata regen job for that file. this should catch legacy files with EXIF rotation that were imported before hydrus understood that info
  • when a file completes a metadata regen, if the resolution changed it now schedules a force-regen of the thumbnail too

  • the rest

  • added a prototype 'delete lock' for archived files to _options->files and trash_ (issue #846). this will be expanded in future when the metadata conditional object is made to lock various other file states, and there will be some better UI feedback, a padlock icon or similar, and some improved dialog texts. if you use this, let me know how you get on!
  • you can now set a custom namespace sort in the file sort menu. you have to type it manually, like when setting defaults in the options, but it will save with the page and should load up again nicely in the dialog if you edit it. this is an experiment in prep for better namespace sort edit UI
  • fixed an issue sorting by namespaces when one of those namespaces was hidden in the 'single media' tag context. now all 'display' tags are used for sort comparison groups. if users desire the old behaviour, we'll have to add an option, so let me know
  • the various service-level processing errors when update files are missing or janked out now report the actual hash of the bad update file. I am chasing down one of these errors with a couple of users and cannot quite figure out why the repair code is not auto-fixing things
  • fixed a problem when the system tray gets an activate event at unlucky moments
  • the default media viewer zoom centerpoint is now the mouse
  • fixed a typo in the client api with wildcard/namespace tag search--sorry for the trouble!

  • some boring multiple local file services cleanup

  • if you have a mixture of trash and normal thumbnails selected, the right-click menu now has separate choices for 'delete trash' and 'delete selected' 'physically now'
  • if you have a mixture of trash and normal thumbnails selected, the advanced delete dialog now similarly provides separate 'physical delete' options for the trashed vs all
  • media viewer, preview viewer, and thumbnail view delete menu service actions are now populated dynamically. it should say 'delete from my files' instead of just 'delete'
  • in some file selection contexts, the 'remote' filter is renamed to 'not local'

• version 439

  • tiled image renderer improvements

  • I believe I fixed the 'non c-contiguous' crash issue with the new tile renderer. I had encountered this while developing, but it was still happening in rare situations--I _think_ in an unlucky edge case where a zoomed tile had the same resolution as the full image rotated by ninety degrees! there is now an additional catch for this situation, as well, to catch any future logical holes.
  • fixed a bug in the new renderer when copying an image to clipboard

  • I greatly mitigated the tiling artifacts with two changes

  • - zoomed in tiles are now resized with a padding area of up to 4 pixels, with the actual tile cropped afterwards, which allows bilinear and lancsoz interpolation to get accurate neighbour data and have gradient math line up with neighbouring tiles more accurately
  • - on resize and zoom, media canvases now dynamically change tile size to 'neater' float/integer conversion dimensions to reduce sub-pixel panning alignment artifacts (e.g. if your zoom is 300%, the tile is now going to have a dimension that is a multiple of 3)
  • I hacked in a 'rescue offscreen media' calculation after any zoom event. now, if the window is completely out of view after a zoom, it'll snap to the nearest borders, lining against them or overlapping into a buffer zone depending on the zoom. let me know what you think!
  • I fixed a PyQt5 specific object tracking bug, I think the new renderer now works ok for PyQt5!
  • cleaned up some ugly code in the resize section that may have been resulting in incorrect interpolation algorithm choice in some situations
  • fixed a divide by zero issue when zooming out tiny images hugely (e.g. 32x32 at 1%)
  • media windows now try to have at least 1x1 size, just to catch some other weird error situations
  • similarly, tile and native sample sizes will have a minimum of size 1x1, which should fix issues during a delayed startup (issue #872)
  • cleaned up some misc media viewer and tile renderer code

  • the rest

  • I started the next round of database optimisation tech, mostly testing out a pipeline upgrade. autocomplete fetching and wildcard file searching for very large queries should be a little faster to cancel now, and in some situations they should be a little faster. they may be slower for very small jobs, but I expect it to be unnoticeable. if you feel autocomplete is suddenly slow and laggy, let me know!
  • I optimised the basic 'ideal sibling normalisation' database query. this is used in a lot of places, so the little saving here should improve a bunch of work
  • I greatly optimised autocomplete sibling population, particularly for searches with a lot of tag results
  • I brushed up the tag import options UI: changed the 'use defaults' checkbox to a dropdown with clear labels for both modes; renamed the 'fetch tags even if' tag import options to 'force page fetch', which is a better description, and added tooltips to describe their ideal use; added tooltips to blacklist and whitelist; and hid the 'load from defaults' button if not set to view specific options
  • added a 'imgur single media file url' File URL Class, which points to direct file links without a referral header, which should fix some situations where these urls were pointed to by other site parsers
  • collections now store the _most recent_ import timestamp of their contents as the aggregate for time imported. previously they had no value, so would sort randomly with each other. collections therefore now sort by time imported reliably with each other, even if there is no 'correct' answer here
  • these new timestamps and service presence generally, and aggregated archive/inbox status, (all of which can update thumbnail display) is now recalculated when files are removed from the collection. so, hitting _right-click->remove->inbox_ will now update collections with a mix of archived and inboxed to remove the inbox icon immediately
  • as the "Retry has no attribute..." network errors have appeared in new forms, I gave the core of the problem another look. we could never really figure this out, but it seemed to be a network version thread safety issue. I think I have ruled this out, and I now believe these may have been occuring during faulty pickling during network session save/load. I fixed the problem here, so with luck this issue will not reappear--if you have had this a lot, let me know how you get on!
  • I broke the requirements.txt into several variants based on platform. we are going to try to pin down good fixed versions of python-mpv and requests/urllib3 for each platform
  • I also updated the 'running from source' help significantly, moving everything to the requirements.txt and making sections for things like FFMPEG and libmpv
  • Also updated the source and contact help around my work style and contact preferences
  • the test.py file now only does the final input() confirmation if there is an interactive stdin to respond

• version 438

  • media viewer

  • I have hacked in tile-based image rendering for the media viewer. this has always been planned as a larger, longer-term job, but the problem of large images is only getting worse, so I decided to just slam out a prototype in a week. if you have a steam-powered GPU or 4GB ram, you might like to wait until next week to update so I can iron out any surprise bugs or performance problems
  • images are now cut into tiles that are rendered on demand, so whenever the image is zoomed larger than the media viewer window, only those tiles currently in view have CPU and memory spent on resizing and storage. as you pan around, new tiles are rendered as needed, and old discarded. this makes zooming in super fast and low memory, even for large images!
  • although I am happy with this, and overall we are talking a huge improvement on previous performance, it is ugly fast code. it may fail for some unusual files. it slices and blits bitmaps around your video memory much faster than before, so some odd GPUs may also have problems. I haven't seen any alignment artifacts (1-pixel thick missing columns or rows), but some images may produce them. more apparent are some pretty ugly tile artifacts that show up between 200% and 500% zoom (interpolation algorithms, which rely on neighbour pixels, are missing border data with my simple system). I will consider how best to implement more complicated but stitch-correct overlapping tiles in future
  • futhermore, a new 'image tile' cache is added. you can customise size and timeout under _options->speed and memory_ like for images and thumbnails. this is a dedicated cache for remembering image resize computation across images and zooms. once you have seen both situations once, flicking back and forth between two images or zoom levels is now generally always instant! this new cache starts at a healthy default of 256MB. let's see how that amount works out IRL--I think it will be plenty
  • I tuned the image renderer cache--it no longer caches huge images that eat more than 25% its total size--meaning these images only hang around as long as you are looking at them--and the prefetch call that pre-renders several files previous/next to the current image no longer occurs on images that would eat more than 10% the cache size. this should greatly reduce weird flicker and other lag when browsing through a series of mega-images (which before would stomp through the cache in quick succession, barging each other out of the way and wasting a bunch of CPU). in real world terms, this basically means that with an image cache of 200MB, you should have slower individual image performance but much better overall performance looking at images with more than about 5k resolution. the dreaded 14,000x12,000 png will still bonk you on the head to do the first render, but it won't try to uselessly prefetch or flush the whole cache any more
  • if you are currently looking at a static image, neighbour prefetch now only starts once the image is rendered, giving the task in front of you a bit more CPU time
  • new options for prefetch delay and previous/next distance are added to 'speed and memory'
  • note this does not yet apply to the old hydrus animation renderer. that still sucks at high zoom!
  • another future step here is to expand prefetch to tiles so the first view of the 'next' media is instant, but let's let all this breathe for a bit. if you get bugs, let me know!
  • due to a Qt issue, I am stopping zoom-in events that would make the 'virtual' size of the image greater than 32,000x32,000

  • account permission improvements

  • to group sibling and parent petitions by uploader (and thus help janitor workflow), the PTR is moving to a system where the public account is download-only and accounts that can upload content are auto-generated in manage services. this code has not been tested much before, and it revealed some very bad reporting and handling of current permissions. I move this forward this week

  • if your repository account is currently unsynced from a serious previous error, any attempt to upload pending data will result in a little popup and the upload being abandoned
  • manage tag siblings and parents will now show service tabs even if the account for those services does not seem currently able to upload tags or siblngs
  • if your repository account is currently unsynced from a serious previous error, this is now noted in red text in manage siblings and manage parents
  • if your repository account does not have sibling/parent upload permission, this is now noted in red text in manage siblings and manage parents. you will be able to pend and petition siblings and parents ok
  • if your repository account does not have mapping/sibling/parent upload permission of the right kind, your client will no longer attempt to upload these content types, and if there is pending count for one of these types, a popup will note this on an upload attempt

  • the rest

  • added https://github.com/NO-ob/LoliSnatcher_Droid to the Client API help!
  • improved some error handling, reporting, and recovery when importing serialised pngs. specific error info is now written to the log as well
  • fixed a secondary error when dropping non-list, non-downloader pngs on Lain's easy downloader import window, and fixed a 'no interesting objects' reporting test when dropping multiple pngs
  • added a 'cache report mode' to help debug image and thumb caching issues
  • refactored the media viewer code to a new 'canvas' submodule
  • improved the error reporting when a thumbnail cannot be generated for a file being imported
  • fixed an error in zoom center calculation when a change zoom event was sent in the split-second during media viewer initialisation
  • I think I fixed an issue where pages could sometimes not automatically move on from 'loading initial files' statusbar text when initialising the session
  • the requirements.txt now specifies 'requests' 2.23.0 exactly, as newer versions seemed to be giving odd urllib3 attribute binding errors (seems maybe a session thread safety thing) when recovering from connection failures. this should update the macOS build as well as anyone running from source who wants to re-run the requirements.txt. I hacked in a catch for this error case anyway, just a manual retry like a normal connection error, we'll see how it goes (issue #665)
  • patched an unusual file import bug for a flash file with an inverted bounding box that resulted in negative reported resolution. flash now takes absolute values for width and height

• version 437

  • misc

  • hydrus now keeps a track of when files were deleted! this information has never been recorded clientside, and it is sadly not retroactively recoverable, but it is stored for all deletes from now on. on occasion, when hydrus says 'this was deleted from xxx', it will now have 'at an unknown time' or a nice '3 days ago' string attached. it will take a few seconds to update this week as the new table data is created
  • the 'trash' panel on review services now has an 'undelete all' button
  • fixed a typo error in manage services when auto-creating a service account when more than one type of account can be created
  • the thread watcher page now sorts the status column secondarily by next check time (previously, equal status would sort alphabetically by subject as a fallback secondary sort)
  • I have renamed some network concepts across the program. before we had access keys, account keys, and registration keys--now we have access keys (secret password for account), account ids (identifier for account that jannies may need), and registration tokens (one-time token used to create a new account). I hope this reduces some confusion
  • reduced some overhead when fetching media results for a search, and when refreshing their tags on major content updates
  • fixed a 'no such table: mem.temp_int_hash_id_1'-style database error state that could persist for 30 seconds or more after certain rare rollbacks
  • fixed the FlipFlip link html in the client api help
  • fingers crossed, I fixed that bad Applications shortcut in the new macOS release
  • fixed a couple more instances of 'pulsing' progress gauges. now they should be blank

  • more efficient updates in sessions with collected media

  • several updates this week should reduce client UI lag when the session contains any pages with a lot of collected media, particularly when you are also running several downloaders (which spam all sorts of content updates across the client)

  • the content update pipeline now tests collections for their files before content processing, and now filters down to process just the updates in a group that apply
  • collections' post-content-update internal data regeneration routine now has more options for fine regen (e.g. no need for tags recalc if the update was 'archive file'), ignores updates for urls and notes (for which it maintains no summary), and only falls back to 'just regen everything' on file location changes
  • the 'selection tags' taglist now retains intelligent memory of its previous selection through collect/uncollect events, which reduces collect/uncollect lag on well-tagged files significantly

  • boring multiple local file services stuff

  • I cleaned a bunch of old hardcoded references to 'my files' and related code. it is not very interesting, but there are a few hundred references to clean up and convert to a system that supports 1-to-n local services, and this week I started hacking away, mostly presentation stuff, labels on menus and so on
  • your 'my files' now has a separate deletion record to the 'all local files' domain. its count shows in 'review services', and for the moment will just be 'all local files' plus the count in trash, but this will become more important when you can have multiple 'my files'
  • behind the scenes re-jiggering means that the deletion record now records deletion time and original import time. delete and undelete transitions are neater as a result
  • logically, files are now generally no longer moved to the trash nor undeleted from there, they instead fall there when they are in 'all local files' but no longer in any local domain, and are undeleted back to a specific service. a bunch of awkwardness is cleaned up, and import/delete/undelete content updates are regeared and ready for multiple local file services

  • a whole bunch of little things have been fixed and changed behind the scenes. I cleaned file service code in general as I went. examples of little things fixed

  • - a 'delete and do not keep a deletion record' action now correctly does not change the cached number of deleted files as reported in review services
  • - the 'clear deletion record and try again' 'remove from trash' component now uses a unified and improved and UI-updating 'untrash' database action, with correct service count changes and UI-side status changes
  • - the 'clear deletion record and try again' action on downloader import queues now handles mixes of actually deleted files and files just in trash more neatly
  • - in the very odd situation that you are looking at a non-local file on 'all known files' and it is then imported using 'archive on import', its thumbnail and metadata now fade in correctly as archived
  • added some unit tests to test the new file delete/undelete transitions
  • cleaned up a bunch of hacky old db SELECT code

• version 436

  • macOS

  • I fixed an issue with last week's Big Sur compatible release where it wasn't finding your old database correctly--it was defaulting to a different location, so without a specific launch command otherwise, it started a fresh db and said 'hey, looks like first time you ran the program'. if you are a long-time user of hydrus, please install and run 436 as usual, it should figure out your old db location correctly as ~/Library/Hydrus without any launch command override needed
  • If you never ran any of the old macOS builds, and you started using hydrus for the first time on macOS last week with the experimental Big Sur compatible build, your brand new database is in a funky location! don't update yet, or you will delete it! You will want to copy your .db files and the client_files folder from inside_the_435_app/Contents/MacOS/db to ~/Library/Hydrus, which should for most people be /Users/(YOU)/Library/Hydrus. feel free to ask for help if you can't figure this out
  • fixed a 'this is macOS' platform check for newer macOS releases, which ensures the 'userpath' fallback is correctly initialised to ~/Library/Hydrus
  • fixed the new macOS github workflow build script to tell hydrus that it is running from inside an App, so it knows to default to the userpath fallback correctly
  • the macOS build now has the old filename
  • it also has the ReadMeFirst.rtf file and Applications shortcut
  • collected the new build-related files in static/build_files, which will likely see more files in future

  • pending tag cache regen

  • two new maintenance tasks are added to the database->regenerate menu--one that forces a recalc of your total 'pending' count as used in the pending menu, and one that recalculates the cached pending tag mappings for storage tags (just like the display one added some time ago, but one layer deeper). the menu entries are relabelled appropriately
  • these routines will be run on database update, and should correct the bad pending menu counts many users discovered last week (the new efficient way that the pending count is calculated exposed some legacy bad cached pending storage mappings entries. we'll see if they come back, or if this is just clearing up bad counts hanging around from ages ago)
  • the quick pending mapping cache regen routines take a little longer to initialise now, but they now clear out surplus tag data, rather than just regenerating the 'correct' tags

  • misc

  • added an experimental setting to _options->tag presentation_ to replace all underscores in tags with spaces. this is just a render rule, so it will only apply in front-facing 'display' contexts (a bit like how siblings work in search pages, but you see the truth in _manage tags_), will consume a little more CPU with big lists, and may result in some duplicate rows, but let's see how it goes. this is basically a quick hardcoded hack until there is a more beautiful solution here
  • in the two 'Duck' dark QSS styles, removed fixed font size on button labels that wasn't scaling on high DPI screens
  • the filename tagging panel now shows parents and siblings correctly on the 'tags for all' and 'tags for selected' taglists. I'd like to show siblings and parents in the file list above in future, but it'll be a bit more tricky to do neatly and without megalag
  • GUGs and NGUGs now report their reasons for not being functional in the downloader selector list and subscription errors. typically this will be a missing url class or an url class missing a matching parser, but more complicated example-url-parsing errors will also be outlined
  • fixed a bug in the client api in the set-cookies call when no cookies are set, and ensured all cookies added this way are saved permanently (before, some could be lost if that domain was not used in network traffic before the next client shutdown)
  • the 'refresh account' button in _review services_ now works on the new async system. it presents errors nicely
  • a repository's current update period is now stated in its review services panel
  • review services now says 'checking for updates in...' rather than 'next update due...', which is more accurate and will matter more with small update times
  • fixed some false positive instances of 'this server was not a tag repo' error in the network engine.
  • the hydrus server now also outputs hydrus specific 'Server' header (rather than some twisted default) on 'unsupported request' 404s and any other unusual 'infrastructure' 4XX or 5XX
  • if the repository updates in the filesystem are lacking some required file information when calculating what to process, the client now queues those files for a metadata regen maintenance job and raises a cleaner error
  • just as a safety measure, if a repository ever happens to deliver a metadata update slice with a 'next update due' time that has already passed, the client now adds a buffer and checks tomorrow instead
  • a new program launch argument, db_transaction_commit_time, lets you change how often the database's changes are committed to disk. default is 30 (seconds) for client, 120 for server
  • altering the repository update period now prints a summary of the change to the log
  • updated the ipfs links in the help
  • updated the main help index.html and the github readme.md with the user-run repo and wiki at https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts

• version 435

  • misc

  • a new macOS build that should run on Big Sur is now ready, it should be attached to this release. it is built on github automatically, and is thanks to hard work from Suika and ReAnzu. I am attaching my old release as well, just in case I messed up somewhere on my end. if you are a macOS user, please try the new App! it will not work on very old macOS like 10.12, but if this works out today for the majority of macOS users, I will be moving to just putting this new build out going forward. I'll add some polish like the readme.rtf and harmonise the filename etc.. too. I'd love to cut the filesize down, but this may not be possible (it is apparently some modern macOS thing where it bundles old and new versions of libraries in the same App so you basically get it twice)
  • the bottom-right corner of the regular media viewer canvas now also shows media zoom
  • the StringSorter object now has a simple 'reverse' sort type
  • the infamous multi-column list 'last column' width calculations are improved: first, dialogs with multi-column lists should no longer judder back and forth a single character's width as you expand the parent window. also, the last column saved size (which is used in dialog relaunch width initialisation) is now snapped to rounded 5-character intervals, which should mitigate various 'fuzzy' reasons for some dialogs to remember a larger or smaller size and grow or shrink one or more characters' width on the next launch
  • the _help->debug->gui actions_ menu has a new entry to reset all multi-column list saved widths back to default
  • the 'edit OR predicate' panel when you shift+double-click an OR predicate now expands horizontally and vertically with the window
  • the 'edit search predicates' list in the 'edit favourite search' panel now expands vertically with the window
  • the client now detects some invalid tag mapping states on tag upload--when a mapping is both current & pending or when it is both deleted and petitioned. these pair-states are mutually exclusive, normally impossible to get to, but one user who nonetheless ended up in this situation encountered an infinite uploading loop to a tag repository (since the tag was already current/deleted, the pending/petitioned status was not clearing correctly on upload commit). now, the upload will be abandoned and an info message put up with the fix
  • added a new maintenance routine to _database->check and repair_ that fixes logically inconsistent mappings. it has a popup dialog when it works and forces a pending count refresh and shows a summary afterwards
  • the routine that counts up total current or pending mappings on a service when the cached number has been reset is now massively faster (from a 30-60s down to less than a second in my dev tests). it now sums the tag autocomplete cache, rather than counting raw tables
  • fixed the BUGFIX option in 'connections' that allows you to disable ssl verification. this will also be extended at a later date to be domain-specific

  • new server stuff

  • a new permission is added to hydrus service accounts--'manage options'. any account with 'manage account types' will get this by default on update
  • any account on a repository with 'manage options' permission will now see 'change update period' in the admin services menu! it launches a time delta control with the current update period and will send the new one up to the server. the client will resync account, options, and metadata immediately, and the server will generate any now-due updates immediately, so you should be able to watch changes occur in 'review services' and the server terminal live. other users will catch up to the new time when they next hit an update. various hardcoded check periods (like how often due updates are checked for and delay-buffered clientside and serverside) are shrunk significantly. the whole system should react to changes better
  • the minimum settable update time is now 10 minutes (the default value remains 100,000 seconds), but I recommend you try larger, say an hour minimum, at least to start. the network generally works more efficiently with higher numbers, and be warned, if you are adding 144 updates a day, there may be bloat problems after a year
  • let me know how this goes, whether you are running a server on a LAN or just a regular user running on one who gets a new update time!
  • the new 'full metadata resync' routine now triggers an immediate metadata update sync and wakes the daemon involved, so it should now happen as you watch
  • fixed the new pause/play buttons on review services to use neutral pause/play icons, not the downloader pause/play
  • brushed up metadata sync status string on review services
  • cleaned misc server and network code
  • cleaned up some old clientside service code
  • the client api now supports wildcard and namespace tags in the file search call
  • client api version is now 16
  • added https://ififfy.github.io/flipflip/#/ , a slideshow engine that now supports hydrus as a source, to the client api page

• version 434

  • network updates

  • the hydrus network version is now 20. update your clients if you want to keep syncing with the PTR! no rush, but if you try to talk to a new server on an older client, or _vice versa_, you will get a polite error message
  • most of the updates this week are for server administrators and jannies. I have reactivated old functions that were broken long ago and added some new features. the rest is mostly code cleanup and refactoring, improved error handling, preparation for the future, and other unexciting (but still important) work that had piled up

  • clientside network/repo stuff for regular users

  • repositories now have separate pause/play for account sync, update download, and update processing. the confusing old 'working' button is replaced with separate pause/play bitmap buttons
  • admins can now attach a message to individual accounts. if you get a message attached, you will see it as a popup message, and on review services, after the next account sync (within a couple of days)
  • a hydrus server may now allow automatic account creation. a button to check for creatable account types and then create a new account is added to 'manage services'. the PTR is likely to move to this sort of system for upload siblings/parents permissions, where getting away from my shared blob mega-account will make it easier to group petitions in the janitor workflow
  • a logical bug in repository update processing order is fixed. I believe this was the source of some bad siblings for new users who did a lot of syncing in one go. there is no efficient re-processing fix available atm (you'll want 'reprocess content' atm), but I plan to split processing into more pause/play for mappings/siblings/parents, which should add more 'reprocessing' tools
  • I cleaned a bunch of the UI code related to networking--in manage services and elsewhere, so there should be fewer bugs, confusion, and UI lag when using these controls. a whole load is still a mess though!
  • fixed the 'see special permissions' button in review services
  • repositories have a new maintenance function, accessed through the new 'reset downloading' button on review services, that force-resyncs all update hashes. this _should_ fix the unusual issue a handful of users had with an extra (invalid/404ing) update hash on the PTR. this function will also auto-trigger on various error states. the reason some users had an extra update hash in storage is still under investigation

  • other stuff, mostly boring

  • fixed an issue with the new file sort asc/desc button where a transition from 'random' to another sort type using a favourite search would always reset the sort order to the top value
  • my asynchronous job object now has a default errback to catch errors more gracefully by default and with special handling in future. clicking an async button in a dialog will now show you the error there and then, rather than just the hidden error popup on the main window
  • added convenience links to the latest build on github to the help menu and html help
  • fixed another place in local file importing where a file that did not pass file import options checks would set 'skipped' status. it now sets 'ignored' like everything else
  • fixed a bug when an 'undelete' call is sent to the media viewer when no media is set (usually during startup/shutdown)
  • I disabled progress gauge 'pulsing' across the program. the way this was first implemented applied too often--I will bring it back to only apply when a job is both indeterminate and currently working
  • my custom button class can now launch its own yes/no confirmation dialogs on click
  • removed a subtag regen routine in the 425->426 update step that was bugging out due to bitrot--it now makes a popup message on boot asking for the routine to be run manually
  • fixed a typo bug in the 'subscription snapshot' debug command
  • misc ancient python 2-to-3 code cleanup
  • updated cloudscraper to 1.2.58

  • admin stuff

  • account modification

  • after a very very long delay, account modification is back. if you have permission to manage accounts, you can directly modify accounts using an account key from the services->admin menu, or for tag repositories you can do it from 'manage tags' from the cog menu on a selection of tags, or for file repositories you can do it from the file right-click menu on an uploaded file, or any petition processing page if you have modify accounts permission
  • the available account modification routines are: change account type, set/extend/clear expiration, ban, unban, set message (new!)
  • the modification routines now print summaries to the server log when they fire, including janny and subject account key
  • the server admin menu now lets you see a list of all the accounts on the service, and also launch the new modification window from it for a selection of accounts
  • the account modification window is also no longer a dialog, but a normal panel that lets you interact with the rest of the program while open
  • the modify UI is now completely async. it loads account information in the background, and all server modification commands and subsequent account refreshing are the same
  • the modify UI now lists separate row summaries, including account type and banned info, for all the accounts it is loaded with. accounts can be checked to apply actions only to a subset. you can also copy the checked account keys
  • the modify UI also highlights if you are one of the accounts being modified, kek
  • the modify UI now shows ban, expiration, and message info, and the 'account info' dict (which is still not great, but can be extended in future), for any listed account when it is clicked
  • I have dropped 'superban' (which deleted all the account's previous content contributions along with the ban) for now. it'll likely return in one form or another in a future account permissions overhaul
  • fixed some dedupe and de-surplus bugs in the 'modify accounts' logic
  • fixed some account ban status presentation logic
  • the UI code has been cleaned up generally
  • retired the old 'get an account's info' admin menu item, since modify accounts dialog now fetches this better
  • hydrus accounts now synchronise twice as fast, every 250,000 seconds

  • account types

  • account types now support optional automatic account creation, default off. it works like a subscription file velocity--'x new accounts per y time delta'.
  • modifying account types now prints summaries, with janny account key, to the log
  • account types now use a newer data serialisation format. the way they are tracked behind the scenes is neater

  • petitions

  • I did not have time to do the petition overhaul. however, I have prepared the network for it, and I have a plan, so I hope to be able in the coming weeks to improve fetch time for petition counts, make a summary list of all petitions, and implement background petition fetching to improve turnaround time
  • the petition UI now has a button to copy the petitioner's account key (e.g if the janny wants to send it to an admin with account modification permission)

  • service options

  • hydrus services now have a separate 'options' object that is synced to all clients alongside account sync. this will be expandable in future without network version hassle
  • the options object now contains stub values for 'server message', 'update period', and 'tag filter'. these values are not yet editable, but I will add them and plug them in in the coming weeks

  • boring network/repo update stuff

  • services now print to log when they host on a port. this applies to the server and the client (for client api and local booru)
  • the repository update routine is now careful to process updates in index order--previously they were unintentionally processed in pseudorandom order, meaning if the processing backlog was long enough, certain add/delete updates of bad content could occur in reverse order. I believe this was the source of some odd persistent bad siblings some users saw after doing a lot of first-time 'catch-up' processing when first syncing with the client
  • the server tracks account_type_keys and account_type_ids in a more careful, service-specific way. all access and modification of account type is cleaned up
  • the AccountType object is graduated to the new serialisation format. the server db stores them in a new table and will convert all legacy objects on update
  • deleted the double-ancient YAML AccountType
  • critical network errors during repository download are now handled a little cleaner and trigger a metadata full resync
  • the server is better about exiting cleanly--last-minute repository metadata changes and very recent session keys are saved
  • wrote a new account account type change routine for the server. it prints a summary, including janny account key, to the log
  • wrote a new account expiration set/extend modification routines for the server. they prints a summary, including janny account key, to the log
  • wrote a new ban routine for the server. it prints a summary, including janny account key, to the log
  • wrote a new unban routine for the server. it prints a summary, including janny account key, to the log
  • wrote a message-set routine for the server. it prints a summary, including janny account key, to the log
  • restored and expanded old server db tests for service creation, account initialisation, account type addition and deletion, registration key/access key/account generation, basic mappings addition, account identification by mapping content, and account modifications: account type, ban, unban, set expires, set message
  • fleshed out and fixed old server unit tests
  • refactored a bunch of server and network code, including to new 'networking' modules, and added a heap of type hints
  • refactored all the server-client variable de/serialisation code, and a bunch of network object code
  • refactored clientside service ui to a new file
  • refactored client network and service panel gui code to new modules
  • refactored the client manage services edit panels to their own, less confusing and coupled, classes
  • refactored the client review services panels to their own, less confusing and coupled, classes
  • cleaned up the clientside service ui, added and migrated to nicer new async functions for server calls
  • cleaned a ton of server and network code in general
  • cleaned up some json/POST variable parsing
  • cleaned admin service menu code

• version 433

  • misc

  • thanks to the effort of a user, this week reintroduces a native twitter downloader! it now gets video! (and 'gifs', which on twitter are just mp4s) please experiment with it and move your nitter subs back to this as you find success. it is called 'twitter syndication' and uses a different access method to get tweet info. it should get the highest resolution videos and images. the search has limited lookup distance, perhaps 500 tweets, but should work for most subscription purposes. this is a first version and may have future updates
  • on the main gui, middle-clicking and left-double-clicking to open the 'new page' dialog, and right-clicking to open the page menu, should now only work on the page tabs or page greyspace. middle-clicking on some random downloader page greyspace should no longer spawn these commands. also, tiny change--middle-clicking now activates here on click _release_ rather than _press_
  • on the file sort widget, the asc/desc sort dropdown is now a 'scrollable' menu button. since sorts are just one of two values, you can now scroll either direction to flip it
  • the 'collect/leave unmatched' in the collect control is also now a scrollable menu button
  • the new tag sort dropdowns are now all scrollable menu buttons. go ham with them
  • middle-clicking the collect-by dropdown now clears it
  • namespace file sort now supports a-z and z-a sorting. files with none of the matching namespaces still count as 'less than a' in a-z terms, but since I am updating all this code, perhaps this could get more attention. I don't use this much, so if you do, let me know what you would like
  • network job controls now show their jobs' current bandwidth limits on their cog menu, split up by network context. you can edit the bandwidth rules directly from this menu, and if it is using defaults, set a specific ruleset
  • userpath generation routines used for database location fallback and default export directory determination now recover from failure in the case of undefined user directory. the client will now not boot if the userpath is needed but undefined
  • the client api no longer prints empty lists for any tag statuses on file metadata calls, nor service entries that have no tags at all
  • fixed the new subscription 'caught up to small initial sync' calculation, which last week was only firing properly after one page of results
  • subscriptions are now better about saving interesting status notes on their gallery logs, rather than overwriting with boring 'no new urls found' messages
  • fixed a typo bug in the new range header implementation for unended ranges

  • boring cleanup

  • refactored ClientGUICommon and ClientGUIControls to a new 'widgets' module
  • refactored the menu buttons from ClientGUICommon to a new ClientGUIMenuButtons
  • wrote a mixin for the basic menu button behaviour and simplified the classes
  • wrote a new 'choice' menu button that has a dropdown checkbox menu and allows mouse scrolling to navigate the list, with wrapping at boundaries
  • refactored the updated network job control to its own file ClientGUINetworkJobControl
  • misc db code cleanup and refactoring
  • misc tag logic cleanup

• version 432

  • tag sorting

  • the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
  • my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
  • when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
  • I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
  • the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
  • fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
  • if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'

  • misc

  • the client now gives a once-per-boot warning popup if your session size exceeds 500k. for those who cannot reduce session size conveniently, this popup can be turned off under _options->gui pages_
  • when the file import options prohibit a file due to filesize or resolution etc.., it should now always record that as an 'ignored' result rather than an 'error'
  • fixed an unusual error popup in thread watcher display that could occur during session load. this problem seems to have been around for a long time, but it required a watcher in a previously saved and still valid 'wait a bit' error state and was only vulnerable for a few milliseconds, so it hadn't come up before. in any case, it is fixed
  • subscriptions with small 'first run' file limits now work better: if you create a subscription with a fairly small 'first run' file limit (this typically matters when the number is smaller than one of the site's gallery page's worth of results), subsequent normal checks with larger file limits will be more aggressive about noticing that they 'caught up' to that small initial sync (previously, they would sometimes incorrectly think the site just got some files tagged out of order and bump right past that initial 'already in db' batch and keep going until they hit their own file limit)

  • advanced string processing

  • added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
  • the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
  • the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
  • the new String Slicer edit UI similarly gets all the strings at that stage of processing
  • future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage

  • server/client api core improvements

  • I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
  • the client now understands 416 ("can't do that requested range m8") errors
  • I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly

  • boring

  • fixed up a handful of typo-borked unit tests
  • fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
  • started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
  • updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance
  • updated sqlite dll on the Windows release to 3.34.1

• version 431

  • misc

  • when parents are hidden in the edit/write taglists (e.g. in manage tags), there is now a '(n parents)' suffix
  • thread watchers that are DEAD or 404 but still have files downloading now report 'working' status until that is done
  • search terms with ';' like 'steins;gate' should now work in downloaders. sorry for the trouble!
  • fixed an issue where un-ideal tags were sometimes becoming non-searchable when they were entirely replaced in manage tags with their 'ideal' siblings (i.e. their autocomplete count went to 0). this was due to overzealous deletion in the new tag definitions cache not filtering out sibling/parent chain members. a small routine will run on update to resynchronise affected tags
  • fixed an issue when loading up files in the main 'import files' dialog where a critical error (as opposed to a nice 'couldn't figure it out, sorry') in mime detection would cause the whole job to hang
  • that main 'import files' dialog now counts 'missing' files separately in the error count
  • fixed tags not updating on the filename tagging dialog when double-clicking to remove from the simple taglists
  • fixed the sort on the manage tags dialog's suggestion taglists--they now preserve their original sort, rather than alphabetising once sibling/parent data is populated
  • the manage parser test panel now catches all network errors. error data back from the server is presented better, and the traceback is now viewable in a special new button on the network job control
  • the edit shortcut set dialog now gives a veto text popup if you try to ok with a shortcut set twice (previously, it would ok and merge down to one command randomly). support for multiple commands per shortcut will come in future
  • entering alt+number in the shortcut entry dialog on windows will no longer spam some errors about 'null character'
  • fixed a 'this object is too huge' check in the database, which mostly affects gui sessions with millions of objects, to check against 1 billion bytes max size rather than 1GB, as here https://sqlite.org/limits.html (issue #816)
  • fixed the 8chan.moe parser, which was pulling hash incorrectly. it should now save more bandwidth
  • updated the e621 parser to pull their (new?) 'lore' tags, which all end in _(lore) and refer to canonical gender and some spice. they come with the 'lore' namespace for now, we'll see how it works out. a user reports these are useful for blacklists

  • new string processing sort step (for advanced users)

  • string processing objects have a new processing step: String Sorter
  • this sorter can sort the whole list of strings, either strict lexicographic or 'human sort' that does numbers properly, asc/desc
  • it can also take a regex for the sort 'key', so you can sample just the number or name you want for sort purposes
  • content parsers no longer have the 'sort formulae results' controls. any content parser with existing sort has been converted so its string processing object has a String Sort step appended
  • the string processing UI is still built around single string processing, so the test UI here is essentially non-functional, but you can see the sort happen in the formula test parse panel
  • I will add a String Slicer in future to sample the list of strings, so you'll be able to grab the top item etc...

  • boring code cleanup

  • refactored the refresh call in filename tagging dialog to nicer Qt signals
  • the add/remove taglists on the simple panel are also moved to Qt signals
  • and so are 'filter' taglists
  • fixed some typos in new help text
  • removed a 'needs restart' string in 'gui pages' options that no longer does

  • here is a hieroglyph falcon

• version 430

  • misc

  • fixed 'unusual character' collapse logic for short text inputs in tag autocomplete lookups. in human, this means typing 'a' now correctly gives you the tag '/a/' and _vice versa_ (issue #799)
  • to make this work, an old database subtag map cache is revived this week in a more efficient form. if you sync with the PTR, it will take a couple minutes to update. the regen routine is also added to the database->regen menu, in case it ever desynchronises in future
  • absent an override referral url, api-linked url fetches now use the original url as referrer. previously they were sending no referrer. this fixes watching spicy boards on 8chan.moe
  • updated a 'get all this stuff' database routine to report more info, and a handful of supermassive jobs (mostly db maintenance regen) now report x/y progress with y, rather than just a nebulous increasing x
  • fixed an odd bug in a common UI text-clearing call that was causing real text not to show up for a while after the clear. this was most apparent in the downloader highlight panels, where status text on file/gallery/network status could sometimes stay blank until a change
  • the manage tags dialog's "there are several things you can do" button box when you enter tags in complicated situations is now clearer. there are several sorts of intro text on the dialog, the button labels are clearer, and button tooltips have more action information
  • fixed the tumblr downloader! sorry for the trouble here, I hadn't realised the situation from some reports. if you have tumblr subs, please go into them and set to 'try again' any recent urls that say 'Found 0 new URLs.'

  • taglists

  • you can now right-click any edit/write taglist (like those across the manage tags dialog) and choose to hide/show the implied parents that now hang underneath tags
  • you can set whether this defaults to hide or show, separately for the regular taglists and the autocomplete results dropdown, under options->tags
  • the taglist now sorts lexicographically using sibling tag data where available. I had expected to make options here to use storage or ideal tag, but once I tried it out, using the ideal all the time felt proper to me, so let's see how it goes
  • fixed the routine that removes mutually exclusive predicates (e.g. system:inbox/archive) when adding to the active search predicates taglist. this fixes the 'exclude xxx from search' menu action and other add/swap actions (issue #815)
  • gave the taglist right-click menu another quick pass. since there are all sorts of actions that may or not appear, and menu items can get pretty wide with tag text, I am trying out an intentionally short and thin top-level menu of 'verbs' that is quick to navigate with your mouse, and then tuck longer and taller stuff in secondary menus

  • boring code cleanup

  • cleaned and unified a bunch of the new taglist sibling and parents display logic and other legacy variables. it now basically all derives from one storage/display state, so behaviour across the program should be more unified. this may cause confusion in some more advanced dialogs, so let me know anywhere it looks weird
  • the 'favourites' autocomplete tab in 'edit/write' a/c dropdowns now show siblings and parents for the current display service
  • the tag suggestions favourites dropdowns and taglists in the options now show siblings/parents according to the current service
  • the 'url class precedence' routine, which tests more 'specific' url classes first when trying to match an url, has a subtle logic change--now, url classes are first considered more 'specific' according to number of path components and parameters that have no default. this stops an url class with multiple optional parameters overriding another with a single fixed parameter (this is what affected the tumblr downloader above). the specific (descending) sort key is now (required components, total components, required parameters, total parameters, len normalised example url)
  • refactored client object serialisation access routines to a new db module
  • refactored database transaction code and status tracking to a separate object
  • refactored some more tag definition routines to the master tag module

• version 429

  • misc

  • fixed a bug in the new taglist backend that would sometimes error out in a paint event(!) on display initialisation or data changes for some clients
  • improved the taglist 'tag' vs 'copyable string' copy/select/action menu logic. e.g. 'namespace:*' is copyable, but it is not a tag
  • thread watchers now skip/clean up unactioned check log entries (this usually happens when a check is due during network traffic paused, queueing the job, and then the client shuts down). if you noticed some odd perpetually 'pending' checkers in last week's status overhaul, this was the issue, and they should clean up. this was always harmless, just revealed with new status code
  • thread watchers now record serious network error detail in the check log
  • thread watchers are quicker about notifying UI on checker log changes
  • thread watchers now report 'time delta' as their simple status when waiting to check, rather that 'checking in (time delta)'. let's see if that fits better in the columns
  • fixed an issue where several dialogs with multi-column lists would reset their 'last column' size to the minimum three characters on the next load if they did not receive certain size events while they were open. you should just have to fix any broken dialogs once and you'll be good again
  • I believe I also improved/fixed the issue of dialogs with multi-column lists sometimes shrinking by a few pixels every open/close
  • the 'we just woke from sleep' detection is now more aggressive. it should now detect a wake after sleeps as short as 60 seconds (down from 5 mins). let's see if we get any false positives during maintenance or other busy periods
  • if you have a complicated database (one stored across multiple locations), the 'database' menu now has a label in place of the simple database's 'backup/restore' commands
  • improved the 'directory is writeable-to' check used in the program. on windows, due to some python tempfile weirdness, this was actually hanging on Program Files.
  • improved the related 'is db dir writeable-to' test in the boot script. if you try to run the program on a custom non-writeable db directory, the crash error should now be nicer, and running a straight client.exe installed to 'Program Files' should now auto-place your db in your user folder, no complaints, like the macOS App
  • corrected 'writable' typo to 'writeable' across the program lmao
  • fixed the new header links in the FAQ file, which I accidentally messed up
  • started work on updating neighbouring .txt tag sidecar export. it isn't ready yet, but it will add tag filters and tag display type to sidecar export with easier expansions in future, and fold it nicely into Export Folders
  • improved some log-off detection + clean shutdown code, but I do not yet have nice multiplatform support

  • filetypes

  • the stacked expand/collapse checkbox widget that lets you select filetypes now always starts collapsed. also, some 'partially clicked' logic is improved when you click through filetype group
  • application/clip (clip studio paint) files are now supported! thanks to a user for helping out here
  • just a side note: I looked into animated webp support this week, but it turns out decoding support is rarer than encoding. my normal and fairly new FFMPEG can't reliably render subsequent frames or figure out duration, nor can PIL or OpenCV. I think we will simply have to wait for an update on one of their ends

  • boring db cleanup

  • wrote a local hashes cache to store hashes for all the files on your disk, much like the tag one. this should speed up all normal searches and other common file lookups in the db
  • the raw storage mapping tables are spun off to their own module
  • basic file info and inbox is spun off to its own module
  • improved and sped up some inboxing file count calculations
  • cleaned up some more misc file metadata and inbox code
  • improved logic in local tags cache

• version 428

  • interesting taglist changes

  • taglists work way better behind the scenes
  • when siblings display with the '(will display as xxx)' suffix, this text is now coloured by the correct namespace!
  • parents now show in 'manage tags dialog' taglists! they show up just like in a write/edit tag autocomplete results list
  • the tag right-click menu has had a pass. 'copy' is now at the top, the 'siblings and parents' menu is split into 'siblings' and 'parents' with counts on the top menu label and the submenus for each merged, and the 'open in new page' commands are tucked into an 'open' submenu. the menu is typically much tighter than before
  • when you hit 'select files with these tags' from a taglist, the thumbgrid now takes keyboard focus if you want to hit F7 or whatever
  • custom tag presentation (_options->tag presentation_, when you set to always hide namespaces or use custom namespace separator in read/search views) is more reliable across the program. it isn't perfect yet, but I'll keep working
  • a heap of taglist code has been cleaned up. some weird logical issues should be better
  • now the code is nicer to work with, I am interested in feedback on how to further improve display and workflows here

  • the rest

  • added two mirrors for nitter, whose main site is failing due to load. I added them randomly from the page here: https://github.com/zedeus/nitter/wiki/Instances . if you have nitter subs, please move their download source to one of the mirrors or set up your own url classes to other mirror addresses. thanks to a user for providing other parser fixes here
  • gallery download pages now show the 'stop' character in the small file column when the files are done
  • gallery download pages now report their 'working' status without flicker, and they report 'pending' when waiting for a download slot (this situation is a legacy hardcoded bottleneck that has been confusing)
  • thread watchers also now have the concept of 'pending', and also report when they are next checking
  • improved the new grouped status sort on gallery downloader and watcher pages. the ascending order is now DONE, working, pending, checking later (for watchers), paused
  • the network request delay after a system resume is now editable under the new options->system panel. default is 15 seconds
  • the 'wait on files too' option is moved from 'files and trash' to this panel
  • when the 'just woke' status is active, you now get a little popup with a cancel button to override it
  • 'open similar-looking files' thumbnail menu entry is moved up from file relationships to the 'open' menu
  • the duplicate filter right-hand hover window no longer has both 'previous' and 'next' buttons, since they both act as 'flip', and the merged button is moved down, made bigger, and has a new icon
  • added 'view next' to the duplicate filter shortcut set, so you can set a custom 'flip between pair' mapping just for that filter
  • thanks to a user helping me out, I was able to figure out a set of lookups in the sibling/parent system that were performing unacceptably slow for some users. this was due to common older versions of sqlite that could not optimise a join with a multi-index OR expression. these queries are now simpler and should perform well for all clients. if your autocomplete results from a search page with thumbs were achingly slow, let me know how they work now!
  • the hydrus url normalisation code now treats '+' more carefully. search queries like 6+girls should now work correctly on their own on sites where '+' is used as a tag separator. they no longer have to be mixed with other tags to work

  • small/specific stuff

  • the similar files maintenance search on shutdown now reports file progress every 10 files and initialises on 0. it also has faster startup time in all cases
  • when a service is deleted, all currently open file pages will check their current file and tag domains and update to nicer defaults if they were pointed at the now-missing services
  • improved missing service error handling for file searches in general--this can still hit an export folder pointed at a missing service
  • improved missing service error handling for tag autocomplete searches, just in case there are still some holes here
  • fixed a couple small things in the running from source help and added a bit about Visual Studio Build Tools on Windows
  • PyOpenSSL is now optional. it is only needed to generate the crt/key files for https hosting. if you try to boot the server or run the client api in https without the files and without the module available to generate new ones, you now get a nice error. the availability of this library is now in the client's about window
  • the mpv player will no longer throw ugly errors when you try to seek on a file that its API interface cannot support
  • loading a file in the media viewer no longer waits on the file system lock on the main thread (it was, very briefly), so the UI won't hang if you click a thumb just after waking up or while a big file job is going on
  • the 'just woke' code is a little cleaner all around
  • the user-made downloader repository link is now more obvious on Lain's import dialog
  • an old hardcoded url class sorting preference that meant gallery urls would be matched against urls before post, and post before file, is now eliminated. url classes are now just preferenced by number of path components, then how many parameters, then by example url length, with higher numbers matching first (the aim is that the more 'specific' and complicated a url class, the earlier it should attempt to match)
  • updated some of the labelling in manage tag siblings and parents
  • when you search autocomplete tags with short inputs, they do not currently give all 'collapsed' matching results, so an input of 'a' or '/a/' does not give the '/a/' tag. this is an artifact of the new search cache. after looking at the new code, there is no way I can currently provide these results efficiently. I tested the best I could figure out, but it would have added 20-200ms lag on all PTR searches, so instead I have made a plan to resurrect an old cache in a more efficient way. please bear with me on this problem
  • tag searches that only include unusual characters like ? or & are now supported without having to lead the query with an asterisk. they will be slower than normal text search
  • fixed a bug in the 'add tags before import' dialog for local imports where deleting a 'quick namespace' was not updating the tag list above

  • windows clean install

  • I moved to a new windows dev machine this week and a bunch of libraries were updated. I do not believe the update on Windows _needs_ a clean install this week, as a new dll conflict actually hits the coincidentally now-optional PyOpenSSL, but it is worth doing if you want to start using the Client API soon, and it has been a while, so let's be nice and clean. if you extract the release on Windows, please check out this guide: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
  • the Windows installer has been updated to remove many old files. it should now do clever clean installs every week, you have nothing to worry about!™

  • boring db breakup

  • the local tags cache, which caches tags for your commonly-accessed hard drive files, is now spun off to its own module
  • on invalid tag repair, the new master tags module and local tags cache are now better about forgetting broken tags
  • the main service store is spun off to its own module. several instances of service creation, deletion, update and basic fetching are merged and cleaned here. should improve a couple of logical edge cases with update and reset

  • boring taglist changes

  • taglists no longer manage text and predicates, but a generalised item class that now handles all text/tag/predicate generation
  • taglist items can occupy more than one row. all position index calculations are now separate from logical index calculations in selection, sizing, sorting, display, and navigation
  • all taglist items can present multiple colours per row, like OR predicates
  • items are responsible for sibling and parent presentation, decoupling a heap of list responsibility mess
  • tag filter and tag colour lists are now a separate type handled by their own item types
  • subordinate parent predicates (as previously shown just in write/edit autocomplete result lists) are now part of multi-row items. previously they were 'quiet' rows with special rules that hung beneath the real result. some related selection/publish logic is a bit cleaner now
  • string tag items are now aware of their parents and so can present them just like autocomplete results in write/edit contexts
  • the main taglist content update routines have significantly reduced overhead. the various expansions this week add some, so we'll see how this all shakes out
  • the asynchronous sibling/parent update routine that populates sibling and parent data for certain lists is smarter and saves more work when data is cached
  • old borked out selection/hitting-skipping code that jumped over labels and parents is now removed
  • 'show siblings and parents' behaviour is more unified now. basically they don't show in read/search, but do in write/edit
  • a heap of bad old taglist code has been deleted or cleaned up

• version 427

  • ghost pending tags

  • fixed another ghost pending tags bug. this may have been new or there since the display cache started, I am not sure, but it shouldn't happen again. it was occuring when a pending tag was being committed to 'current' and another tag in its sibling group already existed as a current tag for that file. the pending tag and its count would not clear for non-'all known files' domains, causing ghosts to appear in search pages but not typically manage tags. you may have noticed ghost tags hanging around after a pending commit--this was it. the true problem here was in the 'rescind pending' action that occurs just before an add/commit. a new unit test tests for this situation both for two non-ideal tags being pend-merged, and a non-ideal tag being pend-merged into the existing ideal
  • wrote a routine to regenerate _pending_ tag storage and autocomplete counts from scratch for the combined and specific display tag caches. this job is special in that it regens tags instantly without having to reset sibling/parent sync. you can run the job from the database->regen menu. this is the start of 'fix just this tag' maintenance ability
  • the pending regen routines will occur on update. it shouldn't take long at all, unless you have five million tags pending, where it could be a couple minutes

  • autocomplete shortcuts

  • there is a new shortcut set under _file->shortcuts_ just for tag autocomplete shortcuts. any 'switch searching immediately' shortcut previously on 'main gui' will be migrated over
  • the tag autocomplete input text box is now plugged into the new shortcut system and uses this set

  • migrated previously hardcoded autocomplete shortcuts to the shortcut system (defaults)

  • - force search now, for when you have automatic searching turned off (ctrl+space)
  • - enable IME-friendly mode (insert)
  • - if input empty, move left/right a tab (left/right arrow)
  • - if input empty, move left/right a service page (up/down arrow)
  • - if input empty and on media viewer, move to previous/next media (page up/down)
  • misc improvements to my shortcut handler
  • misc shortcut code cleanup

  • the rest

  • I fixed a bad example url in the new gelbooru file page parser that was sometimes leading to a link to the gallery url class. this was an artifact of an old experiment with md5-search parsing, now fixed with newer redirection tech. the updated parser is folded into update, and if you ended up with the incorrect link, it should be detected, dissolved, and re-linked with the file page parser
  • thanks to a user report, wrote a new url class for 420chan's newer thread url format
  • sorting a gallery downloader or thread watcher multi-column list by 'status' should now group 'done' and 'paused' items separately
  • fixed a bug in the /add_tags/add_tags Client API call when checking some petitioned tags data types. cleaned all that code, it was ugly (issue #788)
  • added unit tests for /add_tags/add_tags to test the service_names_to_actions_to_tags parameter better and repository actions, including petitioning with and without specified reason

  • code refactoring

  • finally addressing the near-1MB ClientDB file, I have started a framework to break the db into separate modules with their own creation/repair/work responsibilities. this will make the file easier to work on, maintain, update, and test. this week starts off simple, with the master definitions being peeled off into hashes, tags, urls, and texts submodules
  • cleaned some misc code around here, including a bunch of related decoupling
  • ClientDB.py is now in its own 'db' module as well. the db will further fracture and this module will gain more files in future
  • the boot code in the launch scripts is now migrated to the 'hydrus' directory, with the actual launch scripts now doing nicer __main__ checks to not launch the program if you want to play around with importing hydrus. more work to come here
  • finished the help's header linking job--all headers across the help are now #fragment links
  • misc help cleanup

• version 426

  • misc

  • thanks to help from Codexx at 8chan.moe, the old 8kun board is completely migrated and archived at 8chan.moe /hydrus/. going forward I will be maintaining a Hydrus Network General there on /t/ for merged release posts, Q&A, and Bug Reports. the plan is that whenever it fills up, it will be moved to the /hydrus/ archive. the links across the program and help are updated, please let me know if I missed any. Endchan /hydrus/ remains as a bunker
  • fixed a bug where subtag entries in the new tag fast search cache were being deleted for all namespaces when a single namespaced version was went to count 0. it meant some autocomplete results were not appearing, often after some sibling changes. a new 'repopulate' job has been added to the database regenerate menu to fix this efficiently if something like it happens again. this routine will be run on update to fix all users, it shouldn't take long (issue #785)
  • fixed a bug where the new network objects would throw an error on save when a 'dirty' object was quickly deleted. I think this was typically sessions that only have ephemeral session cookies being created in the final five minutes of the program and then being cleared during program exit
  • when an archive/delete filter finishes, it now fires off all its changes in one go. previously they would go in ~64-file chunks over the next few hundred milliseconds. this will add a small amount of 'refresh lag', delaying page refreshes etc..., on bigger filter jobs for some clients, but it will guarantee that if you hit F5 real quick after finishing filtering on a processing page with non-random sort, you won't see the same files again at the top, only for them to be swiftly archived/deleted as you watch. trash file performance is much better these days, let me know how this goes for you if you do megafilters
  • the tag import options whitelist now checks subtags of parsed tags. if you add 'samus aran' to the whitelist, but the site delivers 'character:samus aran', this now passes the whitelist
  • thanks to a user's submission, the gelbooru 0.2.5 post parser is updated and should get tags again, for those users who stopped getting them last week--however, I never experienced this myself, so please let me know if you still have trouble. there could be something more complicated going on here
  • updated the gelbooru 0.2.x gallery parser to handle an alternate form of gelbooru pools--we did not figure out why different users are being given different markup, it wasn't as simple as being logged in or not, but there is a difference for some. this parser is folded in on update, so the gelb pool downloader should be fixed for users who had trouble with it
  • also updated the gelbooru pool gallery url class to infer next page url, as in the alternate form the next page is difficult to parse
  • the 'clear all closed pages' command under the 'undo' menu now asks for yes/no confirmation
  • added a 'callto' profile mode, which will be very useful in diagnosing GPU lag in future. the 'callto' jobs are little off-main-thread things like image rendering and async panel preparation. should help us figure out where big download pages etc... are eating up CPU
  • the different profile modes in the debug menu now all show popup messages, but only when their job exceeds the particular profile's interesting time, usually 3-20ms. this should reduce spam
  • the 'this session' bandwidth tracker on the status bar is now a special tracker that only includes data from boot. previously, it was using the 'global' tracker, which after certain time intervals (four minutes, three hours, three days), will compress bandwidth history into larger time windows to save space. if one of these windows covered time before the client started, it could spookily report a little bandwidth used on a client started with network traffic paused
  • bandwidth data usage in times shorter than the last ten seconds (which are smoothed out to avoid bumps) now also get the 'don't get bandwidth from the future on motherboards that had a briefly crazy system clock' fix from last week
  • fixed some disengaged database tuning that was leading to worse cancel times on certain jobs
  • updated a whole bunch of the help so section headings are links with nice #fragment/anchor ids, making it easy to link other users to a particular section. I will continue this work, and future help will follow this new format
  • fixed some bad character encodings in the changelog document, siblings help, and tagging schema help. these should now be utf-8 valid

  • object load improvements

  • the client now detects serialisable (saveable) objects that were generated in a future version format your client does not yet support. this mostly affects downloader objects like parsers, where you might import an object a user in a much newer version of the client made. for instance, this week some users imported a fixed gelbooru parser in an older client, which was then saved and double-updated later on, and that caused other problems down the line. downloader imports deal with this situation cleanly, but otherwise it mostly makes a popup notifying you of the problem and asking to contact me. there are about 170 places in the program where objects are deserialised and I am not ready to make this a fullblown error until I know more about people's IRL situations. let's hope this is not widespread. if you run into this, please let me know!
  • if you were running an older client and manually imported the updated gelbooru parser that was going around, and then you got errors about 'md5', hex' or 'additional_info' something, it _should_ be automatically fixed on update. you should be able to update from previous to ~422, see it in network->downloader components->manage parsers, and it should just work. many users will have the entry overwritten anyway in the above gelb update I am rolling in. if any of this does still give you trouble, please delete and re-import the affected object(s)
  • importing one of these future-versioned serialised objects using the import/export buttons on a multi-column list, either clipboard, json, or png, will cleanly discard future objects with a non-spammy notification
  • the Lain drag-and-drop easy downloader import does the same
  • the parser 'show what this can parse in nice text' routine now fails gracefully
  • multi-column lists now handle a situation where either the display or sort data for a row cannot be generated. a single error popup per list will be generated so as not to spam, bad sorts will be put at the top, and 'unable to render' will occupy all display cells

  • network server stuff

  • fixed being able to delete an account type in the server admin menu
  • the way accounts are checked for permissions serverside now works how the client api does it, unified into a neater system that checks before the job starts
  • did some misc server code cleanup, and clientside, prepped for restoring account modification and future improvements

• version 425

  • optimisations

  • I fixed the new tag cache's slow tag autocomplete when in 'all known files' domain (which is usually in the manage tags dialog). what was taking about 2.5 seconds in 424 should now take about 58ms!!! for technical details, I was foolishly performing the pre-search exact match lookup (where exactly what you type appears before the full results fetch) on the new quick-text search tables, but it turns out this is unoptimised and was wasting a ton of CPU once the table got big. sorry for the trouble here--this was driving me nuts IRL. I have now fleshed out my dev machine's test client with many more millions of tag mappings so I can test these scales better in future before they go live
  • internal autocomplete count fetches for single tags now have less overhead, which should add up for various rapid small checks across the program, mostly for tag processing, where the client frequently consults current counts on single tags for pre-processing analysis
  • autocomplete count fetch requests for zero tags (lol) are also dealt with more efficiently
  • thanks to the new tag definition cache, the 'num tags' service info cache is now updated and regenerated more efficiently. this speeds up all tag processing a couple percent
  • tag update now quickly filters out redundant data before the main processing job. it is now significantly faster to process tag mappings that already exist--e.g. when a downloaded file pends tags that already exist, or repo processing gives you tags you already have, or you are filling in content gaps in reprocessing
  • tag processing is now more efficient when checking against membership in the display cache, which greatly speeds up processing on services with many siblings and parents. thank you to the users who have contributed profiles and other feedback regarding slower processing speeds since the display cache was added
  • various tag filtering and display membership tests are now shunted to the top of the mappings update routine, reducing much other overhead, especially when the mappings being added are redundant

  • tag logic fixes

  • I explored the 'ghost tag' issue, where sometimes committing a pending tag still leaves a pending record. this has been happening in the new display system when two pending tags that imply the same tag through siblings or parents are committed at the same time. I fixed a previous instance of this, but more remained. I replicated the problem through a unit test, rewrote several update loops to remain in sync when needed, and have fixed potential ghost tag instances in the specific and 'all known files' domains, for 'add', 'pend', 'delete', and 'rescind pend' actions
  • also tested and fixed are possible instances where both a tag and its implication tag are pend-committed at the same time, not just two that imply a shared other
  • furthermore, in a complex counting issue, storage autocomplete count updates are no longer deferred when updating mappings--they are 'interleaved' into mappings updates so counts are always synchronised to tables. this unfortunately adds some processing overhead back in, but as a number of newer cache calculations rely on autocomplete numbers, this change improves counting and pre-processing logic
  • fixed a 'commit pending to current' counting bug in the new autocomplete update routine for 'all known files' domain
  • while display tag logic is working increasingly ok and fast, most clients will have some miscounts and ghost tags here and there. I have yet to write efficient correction maintenance routines for particular files or tags, but this is planned and will come. at the moment, you just have the nuclear 'regen' maintenance calls, which are no good for little problems

  • network object breakup

  • the network session and bandwidth managers, which store your cookies and bandwidth history for all the different network contexts, are no longer monolithic objects. on updates to individual network contexts (which happens all the time during network activity), only the particular updated session or bandwidth tracker now needs to be saved to the database. this reduces CPU and UI lag on heavy clients. basically the same thing as the subscriptions breakup last year, but all behind the scenes
  • your existing managers will be converted on update. all existing login and bandwidth log data should be preserved
  • sessions will now keep delayed cookie changes that occured in the final network request before client exit
  • we won't go too crazy yet, but session and bandwidth data is now synced to the database every 5 minutes, instead of 10, so if the client crashes, you only lose 5 mins of login/bandwidth data
  • some session clearing logic is improved
  • the bandwidth manager no longer considers future bandwidth in tests. if your computer clock goes haywire and your client records bandwidth in the future, it shouldn't bosh you _so much_ now

  • the rest

  • the 'system:number of tags' query now has greatly improved cancelability, even on gigantic result domains
  • fixed a bad example in the client api help that mislabeled 'request_new_permissions' as 'request_access_permissions' (issue #780)
  • the 'check and repair db' boot routine now runs _after_ version checks, so if you accidentally install a version behind, you now get the 'weird version m8' warning before the db goes bananas about missing tables or similar
  • added some methods and optimised some access in Hydrus Tag Archives
  • if you delete all the rules from a default bandwidth ruleset, it no longer disappears momentarily in the edit UI
  • updated the python mpv bindings to 0.5.2 on windows, although the underlying dll is the same. this seems to fix at least one set of dll load problems. also updated is macOS, but not Linux (yet), because it broke there, hooray
  • updated cloudscraper to 1.2.52 for all platforms

• version 424

  • new tag caches

  • as 2020 ended, I attempted but failed to tune fast search for all kinds of clients, big and small and simple and complex. unable to guarantee decent speeds with just code, I have redesigned the tag text search cache. rather than checking the gigantic master table for all namespace and subtag lookups, the client can now zoom in on a small fast cache limited to the current search context, so doing a clever lookup on 'my tags' will no longer be hampered by having PTR beside it, and doing a solid lookup on the PTR or 'all known tags' will no longer be accidentally hampered by an optimisation for another situation
  • the 424 update will take some time to generate the new caches for your existing data. if you don't sync with the PTR, it should be a few seconds. if you do sync, it will be about ten minutes on an SSD (seems about 30,000 definitions a second), and somewhat longer on an HDD. it will count up the tags as it goes, and on the PTR there will be a bit of deletion work, then one or two counts up to perhaps a million, and then one big count up to about 16 million.
  • in my initial tests, this cache adds about 1-2% additional processing time to mass tag changes, but a wide variety of tag lookups and file searches are now significantly faster, have much nicer worst-case lag spikes, and should cancel quicker. these are best in any specific tag domain, although 'all known tags' should still be much better. a future expansion of the tag cache is planned to finally address clean and accurate 'all known tags' searches

  • summary; all these should be faster and cancel faster

  • autocomplete searches for 'subtag*' (most normal searches) are optimised
  • autocomplete searches for 'namespace:*' are optimised, including when the namespace itself is a wildcard
  • autocomplete searches for wildcards with an asterisk in the middle of the subtag are optimised
  • autocomplete searches for wildcards with an asterisk at the beginning of the subtag are optimised (but this is still generally the slowest query)
  • autocomplete searches for namespace and subtag wildcard combinations are optimised, with either or both as a wildcard of any type
  • autocomplete searches for '*' are optimised
  • tag file searches without a namespace (i.e. in file search, with any namespace) are optimised
  • namespace file searches are optimised, including when the namespace is a wildcard
  • wildcard file searches are optimised, for all the classes of wildcard above
  • 'tag as number' file searches are optimised
  • 'has ><= x namespace tags' file searches are optimised for speed, including when the namespace is a wildcard, but still have bad cancelability on large domains. I'll work on this more

  • other tag cache info

  • the 'tag text search cache' regeneration routine under the _database->regenerate_ menu is replaced with a service specific routine for the new cache
  • on boot, if the client sees any of the new cache tables are missing, it notifies you and regenerates the affected subsection of the cache
  • an old method of performing complex wildcard searches was using surplus data and has been eliminated. these searches are now also computationally cheaper beyond the other domain-based optimisations this week
  • I have identified the next bottleneck in the tag search pipeline and have a plan to speed all the above up even further, which can all be done in code
  • thanks to user feedback, I have also identified other wasteful overhead in tag processing. I'll keep working!
  • while the planned 'all known tags' cache will be useful since most file searches are in this domain, it will be a bit of work, so I will first let this new lookup cache breathe for a bit. 'all known tags' will not be nearly as big as the 'all known files/combined file' caches that have hit us with so much CPU recently. I expect it to increase the client.caches.db size by about 5%
  • unified all increments or decrements to autocomplete count caches, no matter the service domain, to one location
  • unified how autocomplete counts are fetched across different service domains
  • optimised specific and combined autocomplete count cache update overhead for new, existing, and deleted tags
  • optimised display autocomplete count cache updates for tags with multiple siblings or parents
  • optimised the 'local tags cache', which does fast tag text fetching for local files, when new tags or files are added/removed from the 'all local files' domain. this now occurs in the same unified autocomplete count update process. it now also caches pending tags that have no current count
  • merged 'exact match' autocomplete tag searching code into generalised wildcard search
  • misc autocomplete and other tag code cleanup and harmonisation
  • ditched some old mass UNION queries that were not cancelling well

  • the rest

  • when you paste queries into a sub, the summary 'these were/were not added' dialog now always appears, and if you paste empty whitespace, it now says so
  • the manage siblings/parents dialogs now specify which services apply which siblings, whether they are fully synced, the current display tag sync maintenance settings, and ultimately whether you can expect changes to apply quickly after dialog ok
  • when a text entry dialog comes with suggestion buttons, it now focuses the text box by default. sorry for the trouble here! (issue #765)
  • updated a couple petition reason suggestions in manage tags and parents
  • added a shortcut to 'main window' to refresh _manage tags'_ related tags suggestions with 'thorough' duration. in future, these dialog-specific actions will be moved out of 'main window', these have just been a 'temporary' patch
  • updated the 'running from source' and 'install' help with some new numbers and info about mpv, and updated the 'server' help with a document helpfully provided by a user explaining that the server does not do what many new users think
  • sped up 'has tags' file searches in certain situations, mostly when there are few if any other search predicates
  • the default e621 parser now pulls meta tags, thank you to a user for providing this
  • the default nitter timeline url classes are updated, thank you to a user for providing this
  • the new little hook that takes 'file:///' off of paths pasted into the filename tagging path text now also normalises the path, so if you are on Windows, the URI's slashes will be Windows-corrected to backlashes. it also now removes wrapping quotes
  • the hydrus logger again correctly restores stdout and stderr after it is closed on program exit (this was disabled for some reason, but fingers crossed it seems fine now!)
  • an issue where automatically started duplicate potentials file search could not cancel when shutdown 'stop work' button was clicked or where idle maintenance mode turned off should be fixed
  • the shutdown maintenance work for the first client shutdown now has a little text saying it is just some quick initialisation work
  • for hopefully the last and completely final time, I think I fixed the invalid tag repair function for certain sorts of tags applied to currently local files
  • improved the way a job thread was pulling new jobs (issue #750)

• version 423

  • tag autocomplete searches

  • the 'fetch results as you type' and 'do-not-autocomplete character threshold' options are moved from _options->speed and memory_ to _tags->manage tag display and search_. they are now service specific!
  • getting the raw '*' autocomplete is now hugely faster when both file and tag domains are specific (i.e. not 'all known xxx')
  • getting the raw '*' autocomplete is now hugely faster in 'all known tags' domain. this is likely still bonkers on any decent sized client that syncs with the PTR, but if you have a small client that once synced with the PTR, this is now snappy
  • the cancelability of 'namespace:*' and 'match namespaces from normal search' searches should be improved
  • 'namespace:*' queries are now much faster in some situations, particularly when searching in a specific tag domain (typically this happens in manage tags dialog) or a small-file client, but is still pretty slow for clients with many files, and I think some scenarios are still bananas. I am not happy here and have started a plan to improve my service domain caches to deal with several ongoing problems with slow namespace and subtag lookup in different situations
  • fixed an issue with advanced autocomplete result matching where a previously cached 'character:sam' result could match 'char:sam' search text
  • some misc code cleanup and UI label improvements in autocomplete

  • the rest

  • the siblings & parents tag menu, which proved a little tall after all, is now compressed to group siblings, parents, and children by the shared services that hold them. it takes less space, and odd exceptions should be easy to spot
  • this menu also no longer has the 'this is the ideal tag' line
  • added 'sort pages by name a-z/z-a' to page right-click menu and tucked the sorts into a submenu
  • the parsing test panel now shows up to 64KB of what you pulled (previously 1KB)
  • the parsing test panel now shows json in prettier indented form
  • when the parsing test panel is told to fetch a URL that is neither HTML or JSON, this is now caught more safely and tested against permitted file types. if it was really a jpeg, it will now say 'looks like a jpeg' and disable parse testing. if the data type could not be figured out, it tries to throw the mess into view and permits parse testing, in case this is some weird javascript or something that you'll want to pre-parse convert
  • the dreaded null-character is now eliminated in all cases when text is decoded from a site, even if the site has invalid unicode or no encoding can be found (e.g. if it is truly a jpeg or something and we just end up wanting to throw a preview of that mess into UI)
  • the 'enter example path here' input on import folders' filename tagging options edit panel now uses placeholder text and auto-removes 'file:///' URL prefixes (e.g. if your paste happens to add them)
  • the 'fix invalid tags' routine now updates the tag row in the local tags cache, so users who found some broken tags were not updating should now be sorted
  • added --db_cache_size launch parameter, and added some text to the launch_parameters help about it. by default, hydrus permits 200MB per file, which means a megaclient under persistent heavy load might want 800MB. users with megamemory but slow drives might want to play with this, let me know what you find
  • updated to cloudscraper 1.2.50

• version 422

  • advanced tags

  • fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here
  • expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again
  • updated the autocomplete result cache object to consult a search's advanced search options (as under _tags->manage tag display and search_) to test whether a search cache for 'char' or 'character:' is able to serve results for a later 'character:samus' input
  • optimised file and tag search code for cases where someone might somehow sneak an unoptimised raw '*:subtag' or 'namespace:*' search text in
  • updated and expanded the autocomplete result cache unit tests to handle the new tested options and the various 'total' tests, so they aren't disabled by accident again
  • cancelling a autocomplete query with a gigantic number of results should now cancel much quicker when you have a lot of siblings
  • the single-tag right-click menu now shows siblings and parents info for every service, and will work on taglists in the 'all known tags' domain. clicking on any item will copy it to clipboard. this might result in megatall submenus, but we'll see. tall seems easier to use than nested per-service for now
  • the more primitive 'siblings' submenu on the taglist 'copy' right-click menu is now removed
  • right-click should no longer raise an error on esoteric taglists (such as tag filters and namespace colours). you might get some funky copy strings, which is sort of fun too
  • the copy string for the special namespace predicate ('namespace:*anything*') is now 'namespace:*', making it easier to copy/paste this across pages

  • misc

  • the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)
  • miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.
  • the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory
  • UPnP errors now have more explanation for 'No IGD UPnP Device' errortext
  • the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work
  • the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes
  • added the --boot_debug launch switch, which for now prints additional splash screen texts to the log
  • the global pixmaps object is no longer initialised in client model boot, but now on first request
  • fixed type of --db_synchronous_override launch parameter, which was throwing type errors
  • updated the client file readwrite lock logic and brushed up its unit tests
  • improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters
  • the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects

  • downloaders and parsing (advanced)

  • started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes
  • added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'Δ"', and the latter does '&'-to-'&'
  • improved my string converter unit tests and added the above to them
  • in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future
  • updated the make-a-downloader help and some screenshots regarding the new hash decoding
  • when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc...)
  • duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs
  • importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs
  • added unit tests for hex and base64 string converter encoding

• version 421

  • misc

  • thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
  • fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
  • fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
  • prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
  • when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
  • the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
  • the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
  • fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
  • I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
  • when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
  • fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
  • reduced overhead of tag filtering, which should improve display speed of taglist for very large pages
  • parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster

  • duplicates search improvements

  • potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
  • the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do
  • all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
  • misc cleanup for duplicates page

  • database modes

  • a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/launch_arguments.html
  • added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
  • ensured --db_synchronous_override was hooked up correctly
  • the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
  • fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
  • db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_

  • technical database nonsense

  • PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
  • hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done
  • hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see
  • in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations
  • when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
  • after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
  • the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
  • improved launch switch code generally
  • boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
  • the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

• version 420

  • misc

  • fixed the bad position indexing when drag-selecting taglists that were scrolled down. this also caused some weird selection when scrolled and clicks included a little mouse movement. sorry for the trouble!
  • ctrl+drag-select now deselects!
  • fetching tag autocomplete results when you have thumbnails and 'searching immediately' on, which has been way too slow recently, now cancels much faster. in some large page situations, it was adding multi-second lag on the first character-press. it also runs faster overall
  • hydrus should now deal better with invalid tags that contain the null character (there is one we know about on the PTR, from a decode of botched Shift JIS, which could crash the client from too many errors during critical paint periods). when a tag like this turns up in a taglist, thumbnail, or canvas background, it now renders as an appropriate 'invalid tag' string, and a one-time 'woah, bad tag, run fix tags now' popup appears
  • regular tag cleaning now looks for and removes null characters, so all new sources of these bad tags should now be eliminated
  • _database->check and repair->fix invalid tags_ now fixes tags with the null character. it also fixes tags so broken that after cleaning they have no subtag left. it also now forces a full media tag reload when it is done for all media
  • the 'regen storage mappings', 'regen display mappings', and repopulate from cache' database routines now have an additional step where you can order them only to work on one tag service, so regenning or repopulating local tags, which usually takes a couple seconds, doesn't need to wait two hours for the PTR to go as well
  • added some menu help to the 'profile modes' debug menu, and gave 'reducing program lag' help page a pass
  • fixed virtual display regeneration on service delete

  • parents and siblings

  • fixed situations where some grandparent and sibling relationships would not appear in the virtual system. it was a bug when certain links of a multi-part display 'chain' were updated at different times. when repopulating chain data, the sibling and parent update routines now correctly chase their complete chains both when wiping ideal data and repopulating from raw data, hitting all levels of the chain, ensuring to go back up and down chains when there are multiple grandsiblings/children/parents, and chasing parents where one or both members have better siblings. thank you to the several users who reported and helped figure out this problem, which was not simple to reproduce (issue #725)
  • your ideal display data will be regenerated on update, which should not take more than a couple of seconds. it will likely correct some siblings and add some grandparents to be filled in by the siblings/parents sync. my PTR test environment went up from about 189,000 display rows to 192,000
  • while sibling and parent lookup is more thorough (and hence more expensive), I also optimised many parts of lookup week. I believe tag display sync and tag processing will be much faster for tags with simple sibling and parent relationships, and slightly slower for tags with complex relationships and many instances to files on your drive. as always, let me know what sort of processing speeds and lag you get, and if you know how to make a db profile, please send them in when it gets bad
  • when a 'write' autocomplete results list includes parent expansion rows (as in _manage tags_), parents now show duplicated and properly for all the tags that have them, including siblings and other children/grandchildren (previously, a parent label could only exist once in a list, which meant parents were ending up hanging off the last valid tag for which they applied)
  • 'write' autocompletes now show results that exactly match the text entry, and all their siblings, when they do not have count but do have sibling or parent data. so, if you type in 'samus aran', and it has a sibling to 'character:samus aran', but 'samus aran' doesn't actually have count, you now get it and all siblings anyway. this may need tuning, but it solves a persistent and annoying lookup and quick-sibling-access problem in _manage tags_
  • copying tags and their indented parents now removes the parent indent whitespace
  • tag sync display now takes way longer breaks (now 30 seconds, was 2.5) between 'normal' background work periods. this thing was hammering people far harder than needed and could clog up db write/commit time and nobble UI responsitivity when big bumps collided
  • the tag display maintenance manager now also tries to detect when many siblings or parents are streaming in (from a migration or a repository process with a heap of data), and pauses work while that continues
  • greatly sped up mass imports of sibling and parent data, either from tag migration or big dialog pastes. what was 40 rows/s should now be about 1,000 rows/s
  • fixed the database menu's 'regenerate tag parents lookup cache', which wasn't hooked up

  • boring changes

  • gave tag parents and siblings update, regen, and chain fetch a full pass, correcting bad queries to fix the above, fixing raw pair chain level navigation and parent-sibling idealisation, and optimised these lookups as well
  • fixed some tag_id vs ideal_tag_id nomenclature (and related bugs) in tag parents cache
  • optimised 'all known tags' autocomplete count fetching a little. tag autocomplete and search should be a bit faster in this domain
  • reduced display sync pre-processing overhead by about 30% with a better random pair sampling routine
  • reduced the overhead of my now very commonly used single integer memory table select optimisation. this now recycles tables after use, which reduces overhead about 50% in small number scenarios. all features of the database will enjoy this speed improvement, particularly small repetitive tag lookup jobs (such as the new display sync and repository tag processing)
  • reduced overhead on some sibling chain lookup code
  • reduced overhead on the sibling lookup used by manage tag dialog taglist
  • reduced overhead on some parent chain lookup code
  • tiny optimisation on single sibling chain lookup
  • sped up the ancient OG single tag->tag-id fetching routine, seems to work about twice as fast now
  • more misc optimisations, mostly list/set/dict comprehension rewriting to reduce overhead, across virtual sibling and parent code
  • added a full combined siblings and parents unit tests for the main missing parent chain link problem solved this week
  • added a full combined siblings and parents unit test for large real world data added in multiple pieces
  • 'a file identifier was missing!' critical errors now print a stack trace to the log for further debugging info
  • updated the 'help my db is broke.txt' document with a couple new comments

• version 419

  • tag lists and editing predicates

  • you can now set the default value for any editable system predicate. a star button beside each panel lets you set or clear the custom default
  • all editable system predicate panels now put 'recent' predicate buttons up top, for the five most entered predicates of the respective types. this is a little jank and grows pretty tall with multi-pred-type panels, but let me know what you think
  • all tag lists now support drag-selection!
  • taglists now have 'open a new OR page' menu entry when more than one tag is selected
  • when taglists can change the current search, they now have an 'add an OR to current search' menu entry when more than one tag is selected
  • OR Predicates are now editable! they launch their own little autocomplete input that is a little jank because you can technically make nested ORs, but it works!
  • system:rating is now editable! it launches the whole stack every time. the stack alignment is messed up though :/
  • invertible predicates (inbox/archive, tag/-tag, etc...) now flip on double-click only if you have one selected. if you have more than one selected, they appear as invertible buttons along with the rest of the edit UI
  • the active search predicates taglist now has an 'edit search terms' menu entry, if you find shift+double-click a pain
  • when you shift+double-click on more than one tag to add them to the current search, this is now added as an OR
  • similarly if you shift+middle-click on more than one tag, the new page is now an OR
  • when editing predicates, edited predicates now stay selected
  • shift+clicking on an already selected tag no longer adds any new selections (i.e. shift+click filling-in). this should make it nicer to do shift+double-click on selections. furthermore, the 'last clicked' focus ghost (from which a shift+click selection cascade starts) on tag lists is now cleared on edits or removes, which should reduce some other crazy/annoying select behaviour here
  • the list of active search predicates now correctly initialises sorted
  • entering hex hashes into system:hash or :similar_to now has unified hash parsing, auto-removes 'md5:'-style prefixes, and presents detailed error information when a hash is too long or short

  • faster and snappier file and tag searching

  • searching for files by complicated wildcard (i.e. a search phrase that includes an asterisk in a non-rightmost character position) is now greatly optimised when the tag does not start with an asterisk (e.g. 'sm*l' is now much faster, '*all' is still hellmode), and now cancels (due to hitting the stop button or changing the query before results come in) much faster thanks to a new unified results fetching and cancel-checking routine
  • rewrote my autocomplete tag search to use the new namespace and subtag lookup code from the virtual siblings and parents system, unifying lookup logic and benefitting from the same new complicated wildcard optimisation and fast-cancel tech
  • autocomplete tag count aggregation (a later step, after the initial lookup) benefits from a little faster cancel tech
  • all file queries based on tag, wildcard, namespace, tag count, and tag existence now use the new fast-cancel tech. if you put in a 'has >4 tags' query and it is taking ages, changing the query or just hitting the 'stop' button should now free up the db pretty fast
  • related tags suggestions also gets the cancel tech and is now more timing precise for tags with either huge or tiny count

  • client api

  • the /get_files/file_metadata call now returns a service_names_to_statuses_to_displayed_tags structure, which reflects the sibling-collapsed and parent-added tags, as displayed to the user in UI. the help is updated to reflect this
  • the client api version is now 15

  • the rest

  • fixed an issue where regenerating the tag definition search cache would not tidy up the 'I am busy' modal dialog once it was done, resulting in a soft lock
  • fixed another upnp error handling bug, this time in the upnp daemon
  • updated Qt to 5.15.2 on Windows and Linux builds. this should fix the unusual button clicking area problem for some custom styles

  • boring specific code changes

  • wrote widgets to edit invertible preds and OR preds
  • pulled the messy rating code out of the rating system predicate ui code to their own widgets
  • wrote some special predicate ui definitions and initialisation handling for OR preds and grouped 'multiple' preds (for ratings)
  • refactored search and predicate ui code to a new 'search' module
  • refactored collect and sort widgets away from search code
  • misc layout improvements for system pred edit ui

• version 418

  • almost all system predicates are now editable if you shift+double-click them! you can also edit several at once in the same dialog
  • if you double-click on any predicate type that is not editable but does have an inverse version (e.g. archive/inbox, has audio/no audio, and tag/-tag), the inverse version(s) will be swapped in
  • all legacy custom system predicate defaults are eliminated this week. the panels now show a fixed default on launch, and will get a flexible favourites system in future, along with 'recently entered' quick buttons
  • restored the 'show system:everything' and 'hide archive/inbox' options, which were inadvertantly hidden when file system predicate defaults were hidden, to the new _options->search_ panel
  • fixed the borked list height for the file viewing statistics system pred panel checkbox lists
  • fixed an issue where namespace:anything predicates would not propagate to new pages on 'open page with these tags' commands

  • boring code specifics

  • updated almost all the system predicate panels to take arbitrary initialisation values, and wrote a 'can I edit this' test for all predicate types to help some finnicky which-panel-and-pred-to-use issues
  • wrote some new filtering code and a little UI to handle editing of system preds
  • cleaned up some of the taglist item activation code

• version 417

  • the hydrus network version is increased this week from 18 to 19. clients and servers can only talk to each other when they are on the same version, so please update your clients if you wish to keep talking to the PTR, and your own servers if you have a home network setup or similar. if a server and client are on different versions, you will get a polite error when they next try to talk, and sync will be paused
  • added 'run all export folders now' shortcut command to 'main window' shortcut set
  • added shortcuts to the 'main gui' shortcut set for navigating the currently selected page. you can move left, right, to the leftmost on the current row, or to the rightmost. the left and right will cycle up a page of pages layer when at left/rightmost boundaries, letting you iterate through all pages in a depth-first manner
  • updated the default newgrounds parser to deal with artists with more than 60/70 items in one art gallery (essentially, some clever 'next page' fetching now occurs to get older info that in your browser is drawn in as you scroll down). if you have some subscriptions for artists where you know this is true, try doing a full reset on them
  • added realbooru to the hydrus defaults. they also apparently just switched away from a gelbooru 0.2.x site, so if you have a gelbooru parser with a realbooru example URL, I remove that example URL
  • updated the page initial media load routine to my new async job
  • updated the imported file presentation page-publish routine to my new async job
  • when drag and drop or import file presentation now wants to add files to a page that is not yet fully loaded (rare, but possible for large sessions), that page now remembers the files it should add and appends them once load is done. these files-to-be-added are also preserved through a session save, if the client is closed before this long-loading page is initialised
  • updated windows mpv, the reported api version is now 1.09
  • updated windows ffmpeg to 4.3.1
  • updated windows release to sqlite 3.33.0
  • updated windows opencv to 4.4.0
  • just a little thing--I took the source links out of the release post. anyone running from source is probably pulling straight from the github repo anyway
  • cleaned up some misc inelegant string code
  • misc other cleanup

  • macOS shortcuts

  • the client's shortcut system now detects macOS-specific 'scroll start/end' states, and will not spam scrolls or errors when these states are held
  • the client's shortcut system now attempts to detect artificial trackpad scroll/wheel events, and adapts the relative speed of scroll event generation according to the respective trackpad velocity. let me know how the hell this works for you in media viewer etc... (issue #710)
  • the client's shortcut system now detects Control and Command as separate and reliable modifiers in macOS, with correct shortcut string rendering (issue #717)

  • upnp

  • fixed the awful typos in the upnp add-mapping error handling I changed last week. I am sorry for this!
  • improved the async mappings and external ip fetch routines in upnp dialog. closing the dialog while a job is going on should now be completely ok
  • upnp dialog add, edit, and delete actions are now async (they won't hang the UI while they work)!
  • all the upnp async jobs should now disable the main list controls while they work
  • fixed the 'edit' action on upnp dialog to correctly remove old and existing mappings depending on what was edited
  • when adding a mapping for an (external_port,protocol) that is already mapped, the upnp dialog now asks if you want to overwrite, rather than just failing with a notification
  • after an async action in upnp dialog, and a mappings refresh triggered, the cached external IP should now be properly restored to the status area
  • pulled parsing code out of upnp code and wrote some proper unit tests for this so stupid typo errors should not happen again

  • parsing

  • subsidiary page parser separation formulae that throw an exception will now be ignored, as if they parsed nothing. in the weird case that you might receive json or html, you can now create subsidiary parsers for both types, and the one that fails will do so gracefully and silently
  • URL Classes now have a key->value 'header override' value. any time one of these URLs is hit, these headers are added!!! be careful with this, but it may solve some tough problems. also, sorry, the URL Class UI is becoming a hellstack, I need to break it into tabs or similar

  • client api

  • added documentation for the new add_files commands, delete_files, undelete_files, archive_files, and unarchive_files
  • added unit tests for the new commands

• version 416

  • misc

  • the new siblings and parents taglist menus now copy just the actual tag when you click, excluding the 'ideal/child/parent:' prefixes
  • added a checkbox to _options->files and trash_ that allows you to automatically prefix hashes copied to clipboard with their hash type in a booru-lookup friendly manner, such as "md5:2496dabcbd69e3c56a5d8caabb7acde5"
  • the media viewer now remembers if it was previously maximised when you set it to un-fullscreen (before, it would always restore-window-ise)
  • fixed the 'test address' button in _manage services_ for hydrus administration services
  • improved the 'add upnp mapping' error handling to better catch 'already mapped' error, with separate errors for redundant, already-on-but-wrong-port, and already-on-another-computer
  • improved error handling when saving objects to the database, particularly for encoding or giganto-size-session errors
  • rewrote my tag sibling lookup unit tests to deal with more situations
  • wrote similar fairly comprehensive tag parent lookup unit tests

  • new downloaders

  • rolling in a user-created thread watcher for warosu. it may be CloudFlare hampered depending on your situation
  • rolling in a prolikewoah thread watcher
  • rolling in a smuglo.li thread watcher

  • multi-column lists

  • spent a bunch of time cleaning out how I calculate multi-column list preferred initial width/preferred current width/minimum width, and made the final column more flexible in its resizing. instances of dialog suddenly getting gigantic because of a final column that wants to size itself at 1,000px should be completely gone, and lists that are shrunk due to non-last-column resizing will now adapt to this situation and not try to flex back to total initial width.
  • multi-column lists now have horizontal scrollbars again for those situations where the parent window is thinner than their (now better calculated) minimum size
  • improved the multi-column list num_rows height calculation, it should have less empty space at the bottom for lists that grow as items are entered into them (such as in the download pages)

  • manage tags megajob speedup

  • sped up manage tags final application step when entering many tags for many thousands of files at once
  • optimised UI-side per-file tag cache (re)generation, reducing overhead and surplus work
  • granularised UI-side per-file tag cache (re)generation based on the four current tag display contexts--now, if a system (e.g. manage tags dialog) only needs storage tags, the different display tags do not need to be regenerated
  • optimised all tag filtering, which is also used in UI-side tag cache regen
  • overall, giganto manage tag dialog jobs should now be faster in several ways. on my dev machine, adding 6 tags to 10k reasonably tagged files went down from 52s to 4.8. even larger jobs will still need a lump of CPU time, but they should scale more efficiently (what was previously O( num_tag_changes x num_total_mappings ) is now O( num_total_mappings ), and better at that)
  • when a huge number of tags is added at once in the manage tags dialog, 'recent tags' is now populated more carefully

• version 415

  • in _options->gui pages_ you can now set the main window's page tab alignment to top/left/right/bottom (previously it was just top/left). this property now updates for all page of pages on options ok, it no longer needs client restart (issue #642)
  • the maintenance task that migrates tag display from the current values to the ideal application now works in significantly smaller steps. big lag from adding hundreds of childen to one parent (or similar siblings) should now be radically reduced
  • rejiggered some layout in the new tag display dialogs
  • added green/red texts to the new tag display dialogs to talk about when sync can work atm and how fast to expect changes to apply
  • reordered the new tag 'siblings/parents info' right-click menu so the dynamic 'has x siblings/parents' submenus are on the bottom
  • added basic client api calls for /add_files/..., delete_files, undelete_files, archive_files, and unarchive_files. they take 'hash' and 'hashes' parameters. I am throwing these out at the end of the week, so they don't have documentation or proper unit tests, but feel free to play with them (issue #393)
  • sped up some UI refresh on content update for very large sessions
  • sped up right-click tag/file menu any/all select actions on very large file sessions

• version 414

  • tl;dr: you don't have to do anything. if you haven't heard of a tag parent before, no worries. the database should work better now

  • top level

  • parents are now completely virtual! this means that when you add a tag parent, the tags that 'fill in' to make it show do not really exist in storage, only in a computed cache. if you decide to undo the parent, the implications are recalculated and the virtual tags disappear, with no permanent changes made. also, petitioning a parent will 'preview' the delete, just as siblings now does
  • siblings and parents are now unified, and the logic is improved. all parents apply to all siblings, so no more worries about retro-active filling-in. the siblings and parents code is now basically 'nice'. this was a lot of quite complicated work, and it solves a number of lingering issues from the original prototypes I made several years ago. I will still do some smaller work and little fixes I am sure in the near future, but the 'big' siblings and parents work is done!
  • like with the recent siblings change, the client no longer needs to do the 'loading parent tags' step when booting--everything is now handled at the db level
  • like with the recent siblings change, you can now edit which services apply their parents to which service, now under _services->manage where siblings and parents apply_
  • in the _manage tags_ dialog (and some other places), tags with parent implications now show a '(x parents)' after their label, much like the 'will display as' sibling suffix. I do not like this, but I ran out of time. I hope to add a more advanced actual listing of virtual tags with a nice 'ghostly' colour or similar in future
  • right-clicking on a tag in a specific tag domain now shows a 'siblings and parents' submenu with detailed info on all known siblings and parents in that domain
  • 'tag' menu entries are moved from the top 'services' menu to a new 'tags' menu. 'pending', when available, is also moved right
  • the process of changing siblings or parents, or which services apply where, is no longer a CPU-laggy process! actual changes, however, may not appear immediately. a maintenance task now tracks what is currently applied and what is 'ideal', and slowly migrates to the ideal in the background in little chunks. in most situations, the changes are very quick, but if you are behind due to big recent changes, they may be delayed. you can manage when this maintenance runs and see the current status under _tags->siblings/parents sync_. this is an entirely new thing, so feedback on IRL work would be appreciated--there may be some kinds of siblings or parents that cause a whole bunch of annoying lag
  • the PTR has a lot of non-virtual parents that were hard-added in older versions over the years. most are fine, but some are like the 'shadow'->'shadow the hedgehog' debacle. now the source of the problem is fundamentally solved, this problem will reduce over time. with luck, before the end of the year, no more will be added at all, and thanks to the janitors, the worst offenders should be chipped away
  • during all this work, a bug with tag siblings and parents repository processing has been revealed (some users do not 'get' all siblings/parents for some reason). now the system is nice and undoable, this will be more easily addressed in coming weeks, with automatic retroactive fixes rolling out to all clients

  • boring details

  • like with siblings, wrote a parents structure object that constructs the parents tree without loops more simply and reliably. it populates a new parents quick-lookup table in the database, for which a full suite of lookup and maintenance methods are written
  • parents and siblings virtual tag presentation is now unified into a single 'display' (i.e. vs 'storage') system with a more granular tag implication algebra (essentially 0-n rows of 'if A is in storage, show B in display' for every tag) that can calculate new and updated display tags and counts without having to do the expensive 'clear-and-regen' that 408-413 used
  • wrote functions to quickly add or remove a display implication to the 'all known files' or specific file service tag display cache
  • migrated all the combined and specific tag display cache update code (add/remove files, add/remove mappings, add/remove sibling/parent, add/remove sibling/parent application, and misc regen maintenance calls) to use the implication system instead of the sibling 'ideal' system (basically moving from 1->1 to 1->n)
  • completely rewrote the complicated 'all known files' cache 'with tags' and 'with and without tags' lookup routines to use much less overhead in general and to use a single, albeit complicated, count-based query that carefully chooses whether to select the 'with tags' and 'without tags' portions using tags or files where available as the primary selector based on existing autocomplete count data
  • replaced all usage of the old ui-side 'tag parents manager' object. as parents pop in virtually and do not need to be bundled intentionally to various content updates, this was mostly just clearing now-surplus code, but for instance in 'write' autocomplete searches, the parents that appear below search results are now generated at the db level on first search, rather than looked-up live in UI time
  • the parents and siblings lookup tables are now split into two views: what the display cache currently holds, and what it ideally should hold. when adding new sibling or parent data, only the fast ideal table is changed
  • a new complicated maintenance function now takes actual and ideal data for a particular unsynced tag, hashes out the implication changes needed to effect a migration, and performs it
  • a new maintenance manager and accompanying db code now track and manage calls to migrate actual to ideal display presentation, and to update UI afterwards
  • as tag display changes are now more frequent, I have made the routine that refreshes tag UI after sibling/parent changes more efficient. tag display now only refreshes for files that have the affected tags in a particular change
  • wrote the UI panel and dialog to show and hurry up current sync status, and all the background hooks for that
  • added 'tag parents lookup' entry to the database 'regenerate' menu. this routine and the 'siblings' variant are now very quick thanks to the new actual/ideal maintenance system
  • updated my sibling unit tests to deal with the new actual->ideal syncing
  • improved the speed of mappings cache updates when deleting files
  • deleted all the old combined/specific 'regen chain' code and the sibling-based 'get sole/any tagged files' search code
  • optimised and generally cleaned a bunch of the new cache code, particularly cutting out overhead for unusual/small situations
  • fixed a counting bug with 'all known files' tag counts when rescinding pending tags
  • fixed a bug in the siblings display code where deleting or pend-rescinding all of the multiple tags that have the same ideal sibling in the same transaction (e.g. if both A and B sibling to C, and a file has both A and B, and you remove them in one manage tags dialog apply) would not remove the current/pending ideal (issue #571)
  • the 'add_siblings_and_parents' parameter on /add_tags/add_tags client api command is now obsolete! the help is updated to reflect this
  • cleaned up just a bunch of db/ui/tag code mate

  • the rest

  • fixed an issue where long-running 'similar files' search was not cleaning its memory use properly as the job was going on, resulting in out-of-mem errors on very large clients (issue #669)
  • thanks to user submission, rolling in a fix for the default pixiv tag search downloader
  • cloudscraper updated to 1.2.48
  • removed surplus executables from linux and macOS builds (win32 upnpc exe was causing anti-virus false positive on mac lmao)

• version 413

  • added 'sort by number of files in collection' file sort type. it obviously only does anything interesting if you are collecting by something
  • when you enter a tag from a manage tags suggested tags column with a double-click, the tag input box is now immediately focused. entering it with a keyboard action does not move the focus
  • wrote a new routine for the 'check and repair' database menu that scans for and fixes invalid tags. this might be some system:tag that snuck in, superfluous unicode whitespace, or some weird website encoding that results in null characters, or any other old tag that has since become invalid. tag translations are written to the log
  • added an experimental 'post_index' CONTEXT VARIABLE to subsidiary page parsers--whenever a non-vetoed post has pursuable URLs, this value is incremented by one. this is an attempt to generate a # 0,1,2,3 series. feedback on this would be appreciated, so I can formalise and document it
  • added 'no_proxy' option for the options->connection page. it uses comma-separated host/domains, just like for curl or the NO_PROXY environment variable. it defaults to 127.0.0.1. in future, options will be added to auto-inherit proxy info from environment variables
  • fixed an error when subscriptions try to publish to a page name when a 'page of pages' already has that name
  • activated some old 'clean url' parsing tech I wrote but never plugged in that helps parsing urls from source fields on sites that start with non-url gubbins
  • fixed the v411->v412 update step to account for a tags table that has duplicate entries (this shouldn't ever happen, but it seems some legacy bug or storage conversion indicent may have caused this for some users). if a unique constraint error is raised, the update step now gives a little message box and does dedupe work
  • fixed an issue where the 'will display as' tag was rendering without namespace when 'hide namespace in normal views' was on
  • fixed a recent character encoding routine that was supposed to filter out null characters
  • fixed some UPnP error reporting

  • _may_ have fixed an odd and seemingly rare 'paintevent' issue when expanding the popup toaster from collapsed state--it may also have been a qt bug, and fixed in the new qt

  • updated qt to 5.15.1 for windows and linux builds. it fixes a couple of odd issues like 'unclicking' to select a menu item (issue #296)
  • added session save to holistic ui test suite
  • misc code cleanup

  • client api

  • wrote a client test for the help menu so I can test some basic functions holistically, hoping to stop some recent typo bugs from happening again
  • did a couple of hotfixes for v412 to deal with some client api url pending bugs. the links in the 412 release now point to new fixed builds
  • fixed an issue setting additional tags via the client api when the respective service's tag import options are not set to get anything
  • fixed a 500 error with /add_tags/add_tags when a tags parameter is an empty list
  • fixed the /manage_pages/get_page_info client api help to show the 'page_info' key in the example response

• version 412

  • client api

  • added Hydrus Web, https://github.com/floogulinc/hydrus-web, to the Client API page. It allows you to access your client from any web browser
  • added Anime Boxes, https://www.animebox.es/, to the Client API page. This booru-browsing application can now browse hydrus!
  • the /add_urls/add_url command's 'service_names_to_tags' parameter now correctly acts like 'additional' tags, and is no longer filtered by any tag import options that may apply. that old name still works, but the more specific synonym 'service_names_to_additional_tags' is now supported and recommended (issue #456)
  • the /add_urls/add_url command now takes a 'filterable_tags' parameter, which will be merged with any parsed tags and will be filtered in the same per-service way according to the current tag import options.
  • the client api help is updated to talk about this, and the client api version is now 14
  • updated client api help to talk about http/https

  • the rest

  • the 407->408 update step now opens a yes/no dialog before it happens to talk about the big amount of CPU and HDD work coming up. it offers the previous 'full' version that takes all the work, and a 'lite' version that applies no siblings and is much cheaper. if you have been waiting on a PTR-syncing HDD client, this should let you update in significantly less time. there is still some copy work in lite mode, but it should not be such a killer
  • the 'manage where tag siblings apply' dialog now has big red warning text talking about the current large CPU/HDD involved in very big changes
  • a bunch of file-location loading and searching across the program has the opportunity to run very slightly faster, particularly on large systems. update will take a few seconds to make these new indices
  • namespace and subtag tag searches and other cross-references now have the opportunity to run faster. update will take another couple of minutes to drop and remake new indices
  • gave tag and wildcard search a complete pass, fixing and bettering my recent optimisations, and compressing the core tag search optimisation code to one location. thank you for the feedback everyone, and sorry for the recent trouble as we have migrated to the new sibling and optimisation systems
  • gave untagged/has_tags/has_count searches a similar pass, mostly fixing up namespace filtering
  • gave the new siblings code a similar pass, ensuring a couple of fetches always run the fast way
  • gave url search and fetch code a similar pass, accounting better for domain cross-referencing and file cross-referencing
  • fixed a typo bug when approving/denying repository file and mapping petitions
  • fixed a bug when right-clicking a selection of multiple tags that shares a single subtag (e.g. 'samus aran' and 'character:samus aran')
  • thanks to some nice examples of unusual videos that were reported as 1,000fps, I improved my fallback ffmpeg metadata parsing to deal with weird situations more cleverly. some ~1,000fps files now reparse correctly to sensible values, but some either really produce 1000 updates a second due to malformation or bad creation, or are just handled that way due to a bug in ffmpeg that we will have to wait for a fix for
  • the hydrus jpeg mime type is now the correct image/jpeg, not image/jpg, thanks to users for noticing this (issue #646)
  • searching for similar files now requires up to 10,000x less sqlite query initiation overhead for large queries. the replacement system has overhead of its own, but it should be faster overall
  • improved error handling when a database cannot connect due to file system issues
  • the edit subscription(s) panels should be better about disabling the ui while heavy jobs, like large subscription resets, are running
  • the edit subscription(s) panels now do not allow an 'apply' if a big job is currently disabling the ui
  • cancelling a manage subscriptions call when missing query logs were detected no longer causes a little error
  • if a long-running asynchronous subscription job lasts beyond its parent's life, it now handles errors better

  • boring details

  • improved a pre-optimisation decision tool for tag search that consults the autocomplete cache for expected end counts in order to make a better decision. it now handles subtag searches and multiple namespace/subtag searches such as for wildcards
  • wrote fast tag lookup tools for subtag and multiple namespace/subtag
  • fixed some bad simple tag search optimisation code, which was doing things in the wrong order!
  • optimised simple tag search optimisations when doing subtag searches
  • polished simple tag search code a bit more
  • added brief comments to all the new cross joins to reinforce their intention
  • greatly simplified the multiple namespace/subtag search used by wildcards
  • fixed and extended tag unit tests for blacklist, filterable, additional, service application, overwrite deleted filterable, and overwrite deleted additional
  • added a unit test for tag whitelist
  • extended the whole 'external tags' pipeline to discriminate between filterable and additional external tags, and cleaned up several parts of the related code
  • moved the edit subscription panel asynchronous info fetch code to my new async job object
  • cleaned up one last ugly 'fetch query log containers' async call in edit subscriptions panel
  • moved the edit subscription(s) panels asynchronous log container code to my new async job object
  • misc code cleanup

• version 411

  • misc

  • fixed the 'system:(like/dislike) rating = x' search predicate string, which was saying 'unknown' rather than 'like/dislike' in several cases
  • fixed a 'current_count' error in the new file search optimisation code for tag searches where the tag did not exist for any files in the domain (i.e. autocomplete count=0). thank you to users for helpful reports here
  • fixed the recent file search optimisation code to handle 'system:time imported' when it was mixed with tags or search predicates that would pre-populate the query file pool with file domain cross-referenced files. sorry for the trouble!
  • the forced delay overhead for table analysis is reduced from 0.1s to 0.02s. whenever many mostly empty tables need to be analyzed (like on first boot shutdown, when it is usually 100+ tables), it now zips by

  • siblings/tag improvements

  • typing a shorthand sibling like 'lotr' into an 'all known tags' 'read' autocomplete - like on a default search page - now reliably discovers and matches text entry to ideal sibling results like 'series:lord of the rings'. this was previously buggy and unreliable--it now allows the match using better db knowledge, even when the merged 'all known tags' services involved disagree on siblings
  • when typing tags into a 'searching immediately' page that has media, the autocomplete count results that only refer to that media will now match shorthand sibling inputs to the ideal result. media-populated tag search now takes a little bump of extra CPU to fetch results (they are now passed through the db to get nice siblings info), so it is also cached for the duration of your typing (previously, the counts were re-computed on every new keystroke, so this should be significantly smoother to work with on large pages even if that first keystroke takes a moment to give results)
  • when typing into a 'write' autocomplete, like in manage tags, the process that promotes the entry text and known siblings to that entry and a potential ideal sibling result to the top of the list is now more sane. it now also only uses results with nonzero count. we'll see how this last change works out IRL
  • when typing into a 'read' or 'write' autocomplete, the pre-search tag insert no longer has sibling insertion/swapping. it was unreliable before, with weird sibling-swapping in the short moment before real results returned. if you have slow results and often quick-add tags into search pages or manage tags, let me know how this works for you
  • the 'additional tags' tag input dialog off the tag import options edit panel now shows the 'will display as' label
  • the 'favourite', 'file lookup', and 'recent' tag suggestion panels now show the 'will display as' label
  • the 'related' suggestion panel, which works on a slightly different system, now shows the 'will display as' label
  • the 'tag suggestions' options panel's 'favourite tags' edit lookup and list now displays 'will display as' labels and correctly finds service-specific siblings in its results (e.g. you type 'lotr', it also finds 'series:lord of the rings')
  • all autocomplete tag filtering should be just that little bit faster as you type
  • filtering cached autocomplete results based on subsequent search text is now faster
  • autocomplete inputs should no longer return 'ghost' results that have no current/pending count when one of the 'include current/pending' buttons is deactivated
  • the new database autocomplete predicate generation routine now checks for 'cancel search' signals, saving CPU time as you type
  • the slow 'regen chains' maintenance tasks now process sibling chains in random order, smoothing out the 500/100,000 progress label, which previously took about 80% of time on the first 20% of ids due to IRL tag distribution

  • the last UI-side siblings work is cleared

  • the UI-side tag siblings cache is no longer used. the sometimes multi-second 'loading tag siblings' step of boot no longer happens!
  • media autocomplete fetches are now asynchronously populated with siblings data via the db
  • the exact-match and sibling 'insert' predicates at the top of pre-load and post-load read and write autocompletes now rely exclusively on db data for sibling matching
  • taglists now present 'will display as' labels asynchronously and are better about updating those labels when the list's underlying tag service changes
  • the taglist right-click menu that shows siblings to copy now fetches that submenu's contents asynchronously from the database
  • the test panel on a blacklisting tag filter now asynchronously fetches tag siblings to test against from the database
  • the actual blacklist tag filter test now fetches tag siblings to test against from the database
  • reworked my custom tag listbox to handle asynchronous text decoration, and unified sibling decoration for media taglists and string taglists
  • updated my old async updater class to be more flexible for different job types, and cleaned the code that already used it
  • wrote a simple class for one-shot async jobs
  • wrote a simple db lookup for UI-side tag sibling chain members
  • wrote a simple db lookup for UI-side tag ideal siblings
  • bunch of misc sibling, db, and ui work and cleanup to make all this work

• version 410

  • general work

  • fixed a bug in the new file service filtering code that was stopping file upload commands to file repositories or ipfs services from sticking
  • fixed an issue with the export files dialog auto-close-when-done function
  • I think I fixed a possible bug in the boot file location repair/recovery dialog sometimes not saving corrected paths on unusual file systems
  • file migration cancel button and shut off timer should work a bit more reliably, more to come here
  • copying subscription quality csv info to clipboard no longer does nice human numbers (you now get 1234, not csv-breaking 1,234)!
  • may have fixed a very rare 'or predicate' error when opening a dialog with a 'read' autocomplete input, like export folder or file maintenance jobs dialogs
  • all pages are better about dealing with missing (i.e. recently deleted) services on load, and autocompletes also
  • error handling from servers with strange character encodings should be better about dealing with null characters
  • cleaned up the combined display regen chain code
  • deleted some obselete db code

  • optimisation review

  • after more profiling, and thanks to additional input from users, I have done another round of optimisation for the new caches. using a new technique, more than just mappings are sped up this week - a number of queries that were prone to lag spikes should now have much more reliable speed and also be faster when hammered often

  • join and analyse db optimisations

  • these are mostly forcing table join orders, which reduces lag spikes, and reducing some related pre-query analysis overhead, which speeds things up more the faster your drive is (up to double processing speed on an ssd). they will affect different clients to different extents, but if your 'related tags' were taking more than a second to load, it should be sorted this week. systems affected

  • archiving files
  • fetching 'related' suggested tags
  • tag siblings regen/update in about ten places
  • all mappings processing
  • additional mappings processing for add/delete, pend/rescind_pend
  • importing or deleting files that have tags
  • loading medias' tags for the first time or on regen
  • loading any media for the first time
  • num notes searches
  • similar files search tree maintenance
  • many general file hash lookups
  • many general tag lookups

  • other optimisations

  • mappings processing
  • sibling processing
  • wildcard tag searches, with and without namespaces, particularly when mixed with other search terms
  • 'tag as number' searches, with and without namespaces, particularly when mixed with other search terms
  • searching for tags when mixed with other search terms
  • has notes/no notes
  • searching files on 'all known files' with general file metadata system predicates (like size, filetype)
  • url class, url domain, and url regex file searches, particularly when mixed with other search terms
  • num tag file searches when mixed with other search terms
  • has/not has tags file searches when mixed with other search terms
  • sped up specific display chain regen significantly, with similar separate current/pending optimisations as last week's for combined
  • converted specific display cache overall regen to use a copy followed by the new chain regen rather than additive file import
  • sped up combined display chain regen a little bit
  • the splash window now updates itself with less UI overhead, so spammy updates (like the new tag regen code) use a little less CPU and fewer UI context switches

• version 409

  • siblings

  • the slowest of the new sibling regen & update code has received a full optimisation pass. some sections take 10% less time, some 90%, and one critical query takes 99% less time. overall, several big jobs work much faster, and ptr processing, which slowed significantly for many users, should be back up to a good speed. uploading pending tags (which tend to be for local files) should be much faster in particular. let's do another round of IRL observation and profiling this week, and I'll keep at it
  • the various 'display' regeneration routines now provide more progress status text, drilling down to the x/y siblings being collapse-counted, or number of files added to a cache, and generally all tag sibling regen got a status update polish pass
  • optimised the way tag sibling application is set--now, only the tag siblings that are changed need to have their counts regenerated. hence, if you just apply (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes

  • the rest

  • fixed the annoying issue with media viewer mouseovers stealing focus/activation from the manage tags dialog. this can now only happen if current focus is on a hover window. sorry for the delay!
  • updated manage tag parents dialog to state the pairs being petitioned on the 'petition reason entry' dialog
  • updated manage tag parents and siblings dialogs to have appropriate 'reason' suggestions for petitions (previously, they were inheriting the same suggestions as for add)
  • ipfs network jobs now have a minimum 'reply' connection timeout of two hours (so giganto directory pushes won't throw an error). connection timeout remains the same, so if the server is hanging on that, it'll still notice
  • fixed the 'test address' button on the IPFS manage services panel
  • petitioning an IPFS file when there is no IPFS multihash entry in the db no longer causes an error. now, in this case, the file entry is removed with no change made.
  • when pending to or petitioning from a file service, a quick filter is now applied to discard invalid files (i.e. (not) already in the service). any weird logical holes where this might occur should now be fixed
  • export folders now catch and report missing file errors more nicely
  • export folders now remember the last error they encountered and report that in the edit export folders dialog

  • boring tag siblings optimisations

  • optimised the tag manager generation routine to use any common file domains for fast cache lookup for any subset of the files available, rather than falling back to 'all known files' domain when there is no single common file domain
  • optimised the new 'all known files' display autocomplete cache to use similar faster specific files cache lookups when available
  • optimised how the 'all known files' display cache regenerates tag sibling chains. it now takes a shortcut when given non-sibling tags and tags where all but one sibling member have zero count, and it can count current and pending counts separately according to the most efficient counting method (e.g. most pre-display pending counts are 0 across the board, so even if current count is a million, the pending count can often be assumed without lookup overhead). furthermore, the 'clever' count has better query planning and less non-sqlite data overhead, and with experimental data is now chosen more carefully. what was previously a 22s job on a test database now takes 5s
  • deduplicated how new mappings are filtered to all the specific cache domains, significantly reducing overhead
  • massively optimised a critical - and the slowest - part of the new 'combined' cache that handles add/pend mappings pre-insert presence testing, speeding up the core query about 100x!
  • reduced some overhead when doing file service_id normalisation in repository processing
  • split up specific chain regen into groups to reduce memory usage
  • optimised specific display tag cache 'add file' updates, and thereby basic cache regeneration, to be just a little faster for files that have multiple sibling tags
  • all predicates made in the database are now populated with ideal and chain sibling information, and this is used for '(will display as xxx)' labels and autocomplete tag search filtering (e.g. you type in 'lotr', it matches an autocomplete result of 'lord of the rings'). there are still some ui-made predicates to figure out, so the old system remains as a fallback
  • related tags lookup is a tiny bit faster and now populates its predicates with ideal and chain sibling info at the db level
  • cleaned up some 'fetch related tags' code, might make it a bit faster for large tag counts
  • cleaned up the way some mapping tables are fetched
  • unified table/table_name nomenclature in the db code
  • updated an old data->ui status presentation method (it typically does stuff like "regenning some stuff: 500/10,000"), to not hog so much UI time and not yield worker threads so often when new statuses are coming in real fast
  • several late optimisations based on IRL testing

• version 408

  • tag siblings cache

  • tl;dr: siblings are faster and better now, you don't have to do anything. some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon
  • wrote the first version of a 'tag display' cache, which stores not your tags as they are, but how they appear after display rules such as siblings, parents, and filtering are applied, meaning this data need not be calculated every time on thumbnail load. this week marks the first concrete step forward in an improvement of siblings and parents storage, and begins with just siblings. all siblings and front-end tags work should be generally faster and more accurate
  • part one is for tag domains cross-referenced with file domains. it maintains virtual sibling-collapsed mappings and autocomplete counts through mappings added, deleted, pended or pend-rescinded, files added/deleted, and siblings added/removed
  • part two is for tag domains on the 'all known files' domain (i.e. no file domain). it maintains virtual sibling-collapsed autocomplete counts through mappings added, deleted, pended or pend-rescinded, and siblings added/removed
  • both parts also support full table drop/regen (under the new database->regenerate->tag display mappings cache) for when my logic inevitably miscounts something. the existing regen 'tag mappings cache'/'tag siblings lookup' commands also regen the display mappings cache, since it relies on them
  • when tag siblings on a repository are petitioned to be deleted, they are now instantly discounted from tag sibling application (previously, they had to be uploaded and committed to count, now both pending and petitioned offer a quick preview of outcome)
  • the display cache supports the tag sibling service application rules under the 'services menu', and regen when that changes, so you can now turn siblings on and off, and apply them across services. as a result, the old 'apply all siblings to all services' option is now gone! as parents will undergo a similar change soon, and the siblings changes this week may lead to some undesired parents in the interim, the 'apply all parents to all services' option is also gone
  • tag autocomplete counts in the form (x-y) due to siblings are eliminated. it will still do it when combining the same merged tag across different services, or when an unnamespaced tag includes how many potential namespaced will also be found
  • the following search types now obey tag sibling application rules accurately: number of tags search, namespace:anything search, wildcard search, tag search (on a per-tag-domain basis, previously it was globally hacked to all siblings), tag-as-number search. for instance, if you search series:anything, a file that has 'metroid' tag-siblinged to 'series:metroid' will now correctly appear.
  • the above search types are now exact to how the tag displays. if you have for files that are tagged 'samus' on either tag service A or B, and service B has a sibling for that to 'character:samus aran', searching for 'samus' gets the results in A, 'character:samus aran' gets the results in B. previously it was an expensive logical mish-mash of 'sure, try and get everything behind the scenes'. now it searches what you see
  • searching for files in the advanced 'all known files' domain currently has no sibling support for the above search types. autocomplete counts should be good, and the results that come up should have the correct tag display, but the actual results are calculated based on storage tags. getting this to work without doubling the size of the db is tricky, so it will have to be ongoing work
  • all tag siblings are now completely virtual. this means that when a tag comes in via a downloader or other means, it will not be automatically coerced to its ideal sibling in the actual db storage tables (the true tags you see in manage tags dialog), but remain as it is. there is no change in sibling appearance in normal operation--it still _displays_ and searches as its ideal sibling. the same will happen to parents in the coming months, and in the interim period, parents no longer apply across siblings. as siblings can come and go from anywhere, they are now divorced from actual stored tag mappings. in a similar way, the manage tags dialog no longer supports the 'hard-replace siblings and parents' command, nor the 'auto-replace with sibling' command. this may be jarring to workflows and preferences, so please bear with me and let me know what feels particularly bad. and please don't worry too much about parents not always being added in the meantime--I hope to do the same transition for them in four of five weeks, and all gaps will be filled in. also, in the coming weeks, I expect to improve manual tagging workflow by indent-grouping edit-view siblings together (ditching the old 'will display as' text) for easier review and selection, a bit like parents. actual 'hard' siblings and parents that do always get irreversibly renamed/added in storage will come in the future as a separate system
  • the 'add_siblings_and_parents' client api parameter no longer adds siblings, and soon will be retired completely
  • I had wanted to completely eliminate the old UI level siblings manager this week, but there are still some systems, mostly tag autocomplete work, that need it and are tricky to swap. I stripped it down, at least, and reduced its update delay to 2 seconds. therefore, the 'loading tag siblings' step of boot still occurs, albiet a _little_ faster. I hope to have it gone soon
  • this is some complicated code affecting core systems. almost everything 'siblings' is now different. there are likely to be laggy parts, awkward new workflow, and possibly some update or miscount bugs as I iron it out. the good news, now they are all virtual, is that problems are undoable. please report any issues, and I will work on it as I keep pushing on this and on parents
  • please expect your client.caches.db file to expand in size about 10-30% or so this week. the update itself will take a few minutes as the improved tag lookups and new caches are regenerated from empty

  • boring mostly db optimisation list

  • after some thought, moved those new options for tag sibling application down to the db. previously, they were stored in an UI object for convenience, but since everything is going down to the db, it is worth doing it properly down there. thus they reset this week to the default
  • I also removed that complicated 'all known tags' page in the tag sibling application options--it wasn't doing enough to justify itself
  • tag siblings lookup cache now obeys the tag sibling application rules and regenerates the appropriate cache when the options change
  • tightened up the db tag siblings lookup cache and wrote more tools for it. it had a couple of blind spots for getting all siblings in a chain. also optimised the lookup for en-masse tag operations
  • tightened up my tag sibling structure builder object, which was not eliminating loops but collapsing them to (generally harmless, but not desireable) (A,A) pairs
  • extended mappings and siblings lookup caches to perform various sorts of tag sibling collapse filtering, to determine files that do or do not have another tag mapping on a tag sibling chain
  • optimised the existing mappings cache in several ways
  • optimised cross-domain file cache mappings filtering, and cleaned the code
  • optimised autocomplete count fetching from the mappings cache, particularly for large result sets
  • optimised how the combined autocomplete count generates from nothing
  • optimised how tags are loaded for search results (thumbnails)
  • optimesed basic tag search
  • greatly optimised how the mappings caches request cross-domain file cache filtering
  • broke up the rescind_pending/add mappings job into simpler separate parts, which was needed for accurate display cache counting. this may end up fixing the other weird pending miscount bug we had
  • the cached 'display' tags are now loaded with regular media results, not generated on the fly on first request (unless in the advanced 'all known files' domain, where it is done quickly on first load at the db level)
  • converted the db over to using its local sibling lookup cache for all sibling jobs
  • all data-level content updates to media result objects now occur in the database loop, reducing lag and eliminating a single UI event loop gap when the objects the UI relies on were desynchronised
  • optimised how the tag and hash id-to-definition cache maintains itself
  • cleaned up cache code generally
  • wrote a ton of unit tests to cover construction, tag, and tag sibling operations on the siblings and display caches
  • wrote a second optimised method for regenerating 'all known files' display cache autocomplete counts from nothing, which, when multiple siblings have wildly different counts (e.g. 50, 100, 200000), instead of counting them all, counts the smaller tags sans the largest, and adds this to the already pre-computed largest count
  • the old ui level siblings manager has been pared down to some final tools that will be trickier to replace

  • the rest

  • fixed the stupid manage tag siblings dialog input/ok bug I introduced last week
  • fixed the pair preview label in manage tag siblings dialog when it asks to enter a reason for a remove petition
  • I believe I fixed the annoying recent bug where the top-right hover window would sometimes not position itself correctly on a window size/move until the top hover was shown once
  • fixed a bug where the 'do you want to do shutdown work?' dialog was not abandoning shutdown if cancelled (rather than yes/no)
  • updated the 'has free space to do db transaction?' checker, which needs to test device partitions, to do two sweeps--first only fast local devices, then potentially mega laggy network discovery if the mount point is not found (hydev was wondering why it was suddenly taking nine seconds to close his test client!)
  • fixed another issue with double-clicking some addremove/queue listboxes when no edit button is set--now in this case they all delete on a double-click
  • fixed a little bad error handling on pending content upload. an error with petitioning certain IPFS uploads is not yet fixed

• version 407

  • sibling prep

  • I am preparing for a new siblings database cache for v408. this will ultimately make siblings (and parents) faster, more accurate, more powerful, and simple to undo. I have decided, as part of it, to make siblings and parents completely virtual (i.e. the tags won't exist for real, they'll be implied). better tools to manage hard-replace siblings and parents will come later, as trying to support both situations at once has not been excellent

  • other sibling work

  • created options to hold per-service sibling and parent preferences, so you'll be able to set up '"my tags" siblings and then "ptr" siblings apply to "my tags"' or 'no parents apply to this service'
  • wrote UI for the sibling options under 'services->manage where tag siblings apply'. you can play with it if you like, and it saves values, but it is not plugged in yet and makes no changes
  • siblings logic is a little tighter. the db and gui side of siblings structure calculation is more unified, petitioned siblings are discounted properly on all generation, and the db side now resolves conflict decisions the same on every regen. the gui-side still runs on an older structure, but will be updated to exactly mirror the db
  • updated and unified how large numbers of raw tag siblings are fetched in the database. it also supports fast tag slicing, speeding up sibling cache maintenance. the siblings lookup cache now uses this method for regeneration and update calls

  • the rest

  • tag right-click menu copying now supports all combinations of selected/all, tags/subtags, and no_count/with_counts where appropriate (issue #325)
  • if the media viewer is too thin for the top hover window to fit into its space, the top-right hover now drops down below it. I don't really like how this looks, and will probably instead figure out a flow layout so the toolbar buttons always fit, but at least they are now accessible (issue #388)
  • altered the above fix--if the top-right hover window can be shrunk to fit in the available space, it will now squeeze in, only bumping down if it can't
  • moving the mouse off an activated (e.g. clicked) hover window now instantly activates the main canvas. this should fix up some fast swallowed clicks and annoying click-to-activate issues with the center-right duplicates hover window, which does not hide (issue #384)
  • the duplicates hover window now positions correctly if its min size is too wide to fit in a thin media window
  • if you make changes to a parser or content parser, there is now a yes/no confirmation when trying to cancel the dialog
  • fixed an issue where 'queue' listboxes with no edit button would throw an error on double-click. now double-click in this case deletes
  • fixed a couple of timestamp convertions that were doing YYYY/MM/DD instead of the more ISO-nice YYYY-MM-DD. also, when in UTC, they'll correctly say UTC now instead of GMT (issue #369)
  • fixed some borked centered text layout on ratings dialog and import folder dialog
  • fixed the manage services dialog's wrong headers for type/name columns
  • added links in the official help to the new user-written simple help guide at https://github.com/Zweibach/text/blob/master/Hydrus/Hydrus%20Help%20Docs/00_tableOfContents.md
  • moved object tag and ratings code to a new client module, 'metadata', and pulled various ratings gui code into a new separate file
  • refactored some more manager code around to generally more sensible locations
  • did a bit more work chasing down the highlight-downloader ui deadlock, which unfortunately still exists
  • reduced the number of db hits some paged downloaders need, particularly on highlight and init
  • updated some test code to support cleverer db testing
  • updated mpv for windows build. api version is now 1.109. this fixes at least one weird linux vm audio driver issue

• version 406

  • subscription management

  • the manage subscriptions dialog now has a 'deduplicate' button. it is enabled whenever your subs of a particular downloader contain duplicate queries. it launches a semi-bananas but thorough 4-step process to ask if you want to do upper/lower-case deduplication, then which downloader, then which queries, then which master sub(s) to retain the queries
  • subscription dedupe within the same sub keeps the query with the most files
  • the manage subscriptions dialog also now has a 'lowercase' button that coerces all queries of the selected subs to lowercase
  • when pasting a list of queries into a subscription, the 'already in sub' test is now caseless. pasting "Samus_Aran" into a sub already with "samus_aran" will not add anything

  • misc

  • url classes now have a checkbox to keep fragment data (e.g. "#kwGFb3xhA3k8B") during URL normalisation. this data is not sent to the server and is not useful for almost all sites, but for sites like Mega, it contains useful clientside javascript navigation or access info if you open the URL in your browser
  • fixed video resolution parsing for some unusual SAR files. this stretches a video slightly, usually when the vid was created or converted with older analog tech (e.g. NTSC)
  • fixed rating system predicate label for 'rated/not rated'
  • the issue where miscounts in pending upload data would persist, sometimes leading to an annoying 'pending (13)' style menu that would not clear without debug action, is now fixed in a cheap way. on any upload action, this cached count is reset. a fix for the actual unusual miscount situation will have to come later
  • the different in-memory manager objects now save changes at different time intervals--lightweight things like favourite searches still save not long after any change, but column widths, network sessions, and bandwidth use now save only every ten minutes
  • I _may_ have fixed an issue with favourite tags not sticking correctly or resetting when added en masse via the tag right-click menu
  • I believe I fixed a rare but permanent ui hang on highlighting a gallery or watcher when that same downloader was spamming through a largely 'already in db/previously deleted' list
  • copying tags 'with counts' now works correctly for simple tag views (previously, it only worked for 'predicate' views)
  • copying tags now preserves the tag order as in the list (previously, it did a human sort)
  • to stop status-sorted gallery and watcher list entries bouncing around so much, they now just say 'working' in their status column when they are working. the highlight panel still reports granular file/gallery info. galleries also now say a more solid DONE when complete, to spot them more easily
  • the gallery and watcher search/checking column now includes stop status in sort
  • fixed the dowloader link in the help to https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders
  • added that same link to the Lain dowloader import panel's help button
  • updated cloudscraper to 1.2.46
  • updated cloudscraper interfacing code to adapt for new reCaptcha->Captcha object names

  • boring code cleanup

  • refactored downloader gui code to its own file
  • refactored network gui code to its own file
  • refactored service gui code to its own file
  • finished import reordering. now all files import in a cleaner order
  • further reworked all hydrus imports to be more breadth-first, loading core modules earlier and catching potential errors in nicer places
  • checkbox selection is now wrapped in the 'quick' dialog system, and all checkbox selections now use this single method
  • simplified and unified a variety of layout code, and fixing some odd layout expanding bugs
  • misc code cleanup
  • deleted some old unused ui code

• version 405

  • tag search

  • system:number of tags now supports namespaces, for example 'find files with two character tags'! (issue #280)
  • it also supports wildcard namespaces, as now do regular namespace search predicates. both run faster. "crea*:anything" is now possible
  • system:number of tags has been optimised, and in many cases is now ten to a hundred times faster
  • system:number of tags still does not support siblings, something I hope to start correcting as of v408
  • both tag existence (numtags =0 or greater than 0) and tag count database routines now respond quickly to 'cancel search' commands, so if you do run a slow query (a bare 'has creator tag' search on 'all known files' on the PTR, for instance), you can now back out quickly after the 'stop' button appears
  • note that 'system:number of character tags greater than 0' and '= 0' are equivalent to +/-character:anything, which will be swapped in if you enter these. also, +/-unnamespaced:anything can now appear
  • the program is a bit better about determining =0 and greater than 0 and less than 1 being 'none' and 'any but none', when it needs to determine optimisations and special labels
  • unfortunately, I am taking away the default value for system:num tags in the options page (edit: I am killing the whole panel now). this old ugly mess of stacked predicate edit panels works on ancient, difficult to update code, so I will retire it and replace it with a unified system that is easy to use, supports in-search system predicate editing, and keeps up with changes automatically
  • system:number of tags is now comfortable with redundancies--if you add >2 and >4, it now knows that >4 is the true lower bound (previously, the one used was random)

  • boring code changes here

  • updated tag existence and tag count searches to take advantage of the tag cache when in a specific file domain (which is pretty much all the time), which should speed them up significantly
  • updated tag existence and tag count searches to more carefully plan their queries, speeding them up both in advantageous and difficult situations
  • cleaned up tag existence and tag count code significantly
  • updated all edit system predicate panels to return full predicate objects, a step towards decoupling them and allowing in-place system predicate editing
  • wrote a new number test object to hold and help with number range test values. num tags now uses it, and eventually all range predicates will too
  • the namespace existence search code ('anything' queries) is now folded into the new generalised tag existence search code
  • streamlined how the search context propagates through all database tag searching--now, most queries do not know or care about domain or current/pending status--they just iterate over n tables as determined by a specialised routine
  • added a handful of unit tests for the new namespace num tag searching

  • database repair

  • the database menu has a new entry, 'repopulate truncated mappings tables', under the newly renamed 'check and repair' submenu, which will try its best to 'fix' a client.mappings.db file that has been truncated due to hard drive fault by repopulating from the local-file-only tag cache. do not run this unless you know you need to
  • the 'help my db is broke.txt' document has a full update pass. the language is clearer, common issues and questions are better addressed, two new recovery routines are added, a section on the stages after boot recovery (like the new repopulate job above) is added, and I added my stock 'now become a backup patrician' nag at the end
  • the debug routine to clear cached service info numbers is now moved to the 'regenerate' database menu. this thing fixes hanging incorrect 'pending' counts until I can fix it properly

  • the rest

  • fixed an issue where when you pasted queries into a subscription, those that were already in the sub (and got the dialog saying so), were being added anyway! I believe this bug came in the last few weeks, after the data storage rewrite. please check your pasted-into subs for dupes
  • fixed tab double middle-click behaviour (so you can spam page close), which I thought I had fixed last week but actually messed up completely right at the end (issue #314)
  • cleaned up some more of the page tab event code--it was a mess all around. should all be on Qt now, no wx hacks
  • network jobs will no longer wait for and consume bandwidth start tokens while all network traffic is paused. all bandwidth competition now halts. (previously, they would continue to consume tokens according to current rules and then all rush to start as soon as traffic was resumed)
  • fixed some client booru/client api requests to correctly 404 on missing file results, rather than 500
  • cleaned up some file sort code and fixed the sort string conversion, which was rendering the opposite sort direction (asc/desc) in summary labels (e.g. on manage favourite searches)
  • cleaned up some ui layout stretching code, including some borked tag import options expand sizing
  • improved some button and padding layout definitions, and improved, slightly, the way the top-right media viewer hover window lays itself out and changes its size on media change
  • improved some review services layout. should be fewer weird heights and widths in unusual situations, and the new multi-column list fits better
  • the manage subs dialog now saves its changes to db more cleanly and atomically
  • updated the default derpibooru parser to pull species tags. ten points if you can guess what that is most of the time

• version 404

  • column lists

  • all multi-column lists across the program now remember the widths of their columns when they are next recreated
  • the last column of any list is now universally the 'stretching' column, which should correctly initialise with its preferred/previous size, but also grows and shrinks with the parent window
  • while all lists retain their initial rows height, and those in the gallery and watcher management panels will continue to grow and shrink in a fixed way, all lists in dialog windows can now be shrunk down to four rows
  • the minimum size of any column is now much smaller, about three characters
  • all column headers now tooltip their name
  • lists should be better about sizing in non-100% OS UI scalings
  • the lists that are automatically sorted (e.g. the download pages, and manage subs) now remember the last sort you gave them

  • future plans, now within reach

  • all lists will sort, sort arrows will appear on the header, and sort will be faster
  • columns will be rearrangeable
  • columns will be hide/showable, and initially hidden complicated columns will be available
  • there will be some, maybe optional, capability to have lists sync live, so if you edit one, the others do the same
  • num rows height memory, maybe--we'll see how the above shakes out first

  • boring code changes

  • moved list code to a new sub-module
  • wrote a status object for column list current columns, widths, and sort, and plugged it into list code
  • wrote a manager to handle column statuses, and plugged it into the main controller and db
  • wrote definitions for every list (66 or so different lists!) and all their columns, and unified width, name, default sort, and future hideabality and default hide/show status to that one easy to edit and extend location
  • rewrote list column and sort initialisation to work off the new status object and added hooks for list sorting and column resizing to save new status back
  • rewrote every list column instantiation to use the new system
  • numerous misc column list code cleanup

  • the rest

  • double middle-clicking on the page tab bar should now correctly close two tabs in a row (rather than opening the rename page dialog on the second)
  • entering an odd number of hex characters into system:hash no longer causes an error. this will be changed in future to properly highlight and explain badly pasted or incorrect-length hashes in future
  • the new red text for non-functional status texts in review services now properly re-colours itself between normal/red when an error or resolution occurs while the panel is open
  • hydrus now knows if it is running in the Haiku operating system and has preliminary platform specificity. if you are interested in helping to get hydrus running properly in Haiku, please join in with github issue #358
  • cleaned up a mix of smaller code, unused variables and imports and so on

• version 403

  • shortcuts

  • shortcuts have a backend update this week. a bunch of hacky stuff is now cleaner behind the scenes, and the related UI has some cleanup as well
  • converted all 100-odd simple shortcut commands from hacky text ids to a proper enumerated id system, and across every single instance across the program
  • wrote nicer descriptive labels for all simple shortcuts. gone is 'focus_media_viewer', now is 'keyboard focus: to the media viewer'
  • if you have no like/dislike or numerical services, the respective application command edit panels now say so and do not allow an ok action
  • like/dislike rating sub-panels now start with 'like' checked
  • when a like/dislike or numerical rating sub-panel is set to 'remove', the action dropdown is set to 'set' (rather than flip) and is disabled, as is the numerical slider
  • application commands now state better "3/5" information about rating actions, rating than the underlying "0.6" float implementation
  • all application commands existing in shortcuts or elsewhere are updated to the new enumerated id system
  • refactored ApplicationCommand (the side of shortcuts that holds the actual action to be done) and its edit UI to new separate files
  • completely refactored the application command edit panel, pulling the simple/tag/rating sub-panels into their own decoupled classes, simplifying the tangle and permitting easier future expansion
  • rearranged some application command functions and contant definitions to more appropriate locations
  • improved how application commands are interrogated by the objects that process them
  • added plenty of type hinting around application command processing code
  • cleaned up a bunch of shortcut and application command code, including some wx->Qt updates as well

  • menu and UI cleanup

  • removed an old wx hack that prohibited last-second ui updates. the exit splash screen now reports final db shutdown info
  • if a service or account is currently non-functional (e.g. all repositories are paused), the appropriate status text is now in red
  • if there is work to do the first time a duplicate page is opened or looked at, it now moves to the 'preparation' tab
  • doing a 'migrate database' file migration now temp-closes the migrate db dialog and hides the main gui while it goes on
  • brushed up the tag filter ui a bit--now only one of the tag_filter/blacklist test phrases only show up, in the appropriate context, and the test text input now supports multiple newline-separated tags (e.g. if you want to paste a bunch)
  • every panel on review services now has a refresh button to force an update
  • the 'clear trash' button on the trash review services panel is now disabled when there is nothing to clear
  • updated edit subscription panel to point to the main html help and brushed up that help to talk about file limits more, also the earlier downloader help has a little section to highlight subscriptions and their use
  • reworked the 'restore from db backup' command--it is now integrated into client shutdown proper, and reports its basic restore progress to the exit splash screen
  • reorganised the 'network' menu. manage subs is now up top, downloader submenus are now split better into high-level vs component-level, and login stuff is pulled to its own submenu
  • put 'network traffic' at the top of the network->pause menu
  • rearranged some of the 'gui' and 'gui pages' option pages and tucked everything into box sections for clarity
  • the search pause/play button on search page tag autocomplete now has a simpler 'search paused' label when paused. the code has a similar nomenclature change, and eventually this will turn into a simple pause/play icon button or similar
  • fixed some weirdness with floating autocomplete dropdowns sometimes not appearing on dialogs on first load
  • fixed some focus logic so set-focus calls on downloader pages should work again on the query input text box and elsewhere
  • unified all numerical rating->stars and stars->rating calculations across the program. this may have fixed some edge-case bugs
  • unified all rating string generation across the program

  • the rest

  • the disk cache options under _options->speed and memory_ are now default off and force-set off for all users on update. as more users are on decent ssds where these options are of limited value (and sometimes negative value), I now only recommend them for users on HDDs
  • added two options for autocomplete results list height to 'gui pages' option page, under the new 'controls' section
  • fixed a critical issue where the client api could duplicate-add tags with url imports to multiple services. the potential service duplicate cascade order was pseudorandom and particular to a client. thanks to a user for figuring this issue out (issue #317)
  • added a 'tag whitelist' to downloader tag import options. its edit button is below the blacklist. when there are no tags in the list, it does nothing, but if tags are added, then files that do not have at least one of the given tags at the download source will not be imported. for instance, if you have a username-based downloader (where you can't add more tags to the query to filter serverside), and you only want their metroid content, you can now filter it simply hydrusside (issue #279)
  • if you are both in advanced mode and a mad lad, the basic blacklist tag filter now allows you to show the 'whitelist' and 'advanced' panels again, if you have a complicated blacklist to set
  • the local booru and client api now support the same https as the hydrus server, using self-signed certificates stored in the db directory. just set the checkbox in manage services and you should be good. self-signed certificates are free and will work on a server hosted off an IP address, but they are imperfect. they are also likely to require special permission to be accepted by the web browser or whatever you want to talk to the https service. however, if you host your client from a real DNS domain and have your own fully signed cert+key files, you can swap them in no problem
  • local booru and client api urls adjust scheme for the new option, and unified and cleaned up how booru share urls are generated internally
  • the way cert+key files are generated is moved from server code to common hydrus code
  • cleaned up how additional db files like certificate files and the mpv conf are managed for backup/restore operations
  • cleaned up some ancient http urls to https. mostly stuff like the regex tutorial links
  • when files are appended to a regular search page (e.g. from a subscription publish to an existing page, or from a mouse drag and drop), the search context will now pause. this is to stop accidental F5 or mass refresh signals wiping out the changed page
  • to break advanced-case gallery search loops, gallery url jobs now have a 'run' identity token. galleries pass their token down to 'next page' or 'sub-gallery' urls they generate, meaning all urls of a particular search run share the same url. gallery logs now ignore to-be-added urls that already exist for their token, terminating loops. new tokens are generated if a search is restarted or similar, meaning duplicate urls can exist in a gallery log, just not from the same starting point (issue #302)
  • improved simple gallery url deduplication in several stages of the downloader pipeline
  • when right-clicking on multiple thumbs, the info lines off the top menu item now list the files' combined viewtimes (this previously only showed when one file was selected)
  • fixed some error reporting problems with adding urls to import via the client api--some url class exceptions were being converted from 400 to 500 errors unintentionally
  • a new stylesheet, 'Hydracula', is added to the default install. check it out under options->style. thank you to a user for contributing this
  • subscriptions are better about calculating a 90 second forgiveness window for bandwidth rules. they should schedule and startup more effectively, and the edit subscriptions and single edit subscription panels should also no longer show bandwidth delays below the next 90s, which are often a technical situation of regular work breaks that are better ignored for the purposes of the dialog
  • went back up to pyinstaller 3.6 again on windows, as 3.5 caused its own Qt bindings dll problems. if you had trouble with 3.6 (401), let me know how this works for you, as there are additional dll-finding fixes included (issue #329)
  • fixed an issue where under some conditions, file save dialogs were only happy with filenames that already existed (issue #319)
  • fixed an issue with the 'client already running' system sometimes not closing the client process correctly when told to cancel the boot
  • bumped the 'space needed for vacuum' estimate up to 120% (was 100%) of estimated final file size, just to catch some edge cases
  • rolling out updated danbooru parsers that pull associable urls correctly, thank you to a user for this fix
  • rolling out an updated deviant art parser that finds some unusual file urls when other methods fail, thank you to another user for this fix (issue #295)
  • upgraded cloudscraper to 1.2.42
  • improved some type hinting
  • fixed up some unit tests for new command and rating data

• version 402

  • in many situations--such as a search result that gives no results, or a search cancel, or a downloader page cleared of a highlight--pages will now report a special status text rather than '0 files', such as 'no results for this search' or 'search cancelled!' (issue #277)
  • new pages, and the first page of a loaded session, should now correctly publish their status text to the status bar immediately after initialisation, (previously blank until first change)
  • clicking the 'searching immediately' button while a search is ongoing now correctly cancels a search, cleaning up status and page and buttons, rather than just stopping current work immediately
  • added 'copy_xxx_hash' shortcuts to the media shortcut set for 'md5', 'sha1', and 'sha512'
  • when copying file hashes to clipboard, a popup appears for two seconds to verify what happened
  • when copying file hashes to clipboard, recovery from missing hashes is more graceful, with multiple error report states
  • the way the client shuts down is untangled. the order in which the gui, managers, threads, database are shut down is smoothed out, with better error handling and fewer potential logical holes
  • the 'should I do shutdown work?' dialog is now only presented in the clean shutdown pipeline
  • menu labels now elide at 128 characters, extended from 64 previously. hopefully this strikes a better balance between fixed texts we do want to read while still not letting long dynamic texts go nuts (issue #276)
  • gallery and watcher pages now have 'show file/gallery log' on their menus, which directly zoom in to the edit dialogs for the top-most selected query or watcher (issue #256)
  • when file maintenance is forced to run from the thumbnail menu or file maintenance job panel, it now provides x/y progress text and gauge based on total jobs, e.g. 1,234/10,000, rather than out of the 256-job batches (issue #264)
  • the simple downloader page now updates its pending jobs list more efficiently, and supports multiple selection, and presents a yes/no confirmation on delete
  • most lists with clipboard/png import/export buttons can now also do .json files. they also accept json files in a drag and drop. you can mix json and png files in a multi-file drag and drop
  • when selecting a parser for a url class in 'manage url class links', those parsers with example urls that match the url class are now separately listed at the top of the choice dialog
  • in the recent autocomplete rewrite, the hidden repository update file domain was accidentally exposed in the file domain button. after some testing, it actually works(!), but as this is an advanced topic, it is now hidden behind advanced mode
  • the way services are deleted or completely reset is now changed to what should be a significantly faster and smaller operation
  • the latest user-made nitter/twitter downloader is rolled in to the update. some little fixes and adds support for mobile.twitter.com url imports
  • fixed an issue where uninitialised repositories thought they were caught up
  • to reflect that it does nothing in this case, the mouse shortcut edit panel now disables the press/release choice on double-click or scroll
  • fixed file save dialogs not filling in the default filename properly
  • removed an old wx safety hack where new pages would silently not create while the client was minimised. this fixes issues with large session loading and subscriptions publishing files to page names that do not yet exist while the client is minimised
  • removed an old wx safety hack where some tag lists would not regen their current tag display while the client was minimised
  • in lieu of a future better bit of html subscription help that I link to from the subscription panel, the 'file limits' help button has temporarily briefer text so it doesn't make such a giant popup
  • moving back to pyinstaller 3.5 (from 3.6) for the windows build, which appears to fix some dll loading for some users (issue #244)
  • the windows and linux builds are updated to Qt 5.15 (from 5.13.2). it does not seem to have the odd problems 5.14 gave us. let me know if you have any trouble or if any weird graphical issues magically fix themselves

  • client api

  • the /get_files/file_metadata call has a new true/false parameter, 'detailed_url_information', default false, that adds 'detailed_known_urls' structure to list the known urls results as in /add_urls/get_url_info. it has a help example and a unit test and everything (issue #235)
  • the client api version is now 13

  • boring cleanup details

  • reshuffled the shutdown code. now the controller takes the lead, booting splash as appropriate and commanding gui to save and close, and then proceeds to other shutdown
  • fast and normal shutdown code is unified, just run differently
  • shutdown calls should now always be idempotent
  • a catch for some OS-level shutdown commands, normally user log-off, also hooks into the newer UI-free fast shutdown
  • SIGINT and SIGTERM also hook better into the new shutdown, and are thread safe
  • performing multiple SIGINTS on shutdown should no longer throw an error after the gui is deleted
  • more potential startup/shutdown errors are now caught and presented to the user and saved to log, with subsequent shutdown urgency accelerated afterwards
  • critical errors on a fast shutdown no longer present to the user--they just save to log
  • updated how an emergency shutdown state is tested
  • updated how a 'clean exit complete' state is set and tested
  • various unusual shutdown states now skip human interaction and jump straight to guaranteed fast shutdown
  • refactored splash window to its own file
  • wrote a new qlistwidget subclass to do some common data storage/retrieval/selection. it will eventually replace most lists across the program
  • the 'queue' list widget that has up/delete/down and add/edit buttons beside a list has nicer backend code and now initialises with its buttons correctly disabled due to no selection
  • the similar 'add/edit/delete' list widget is updated to use the nicer backend
  • some wx->Qt list hacks, which were themselves using borked old display-string-based indexing, are deleted
  • the repository download/process daemon has been moved to the newer job scheduler. it should start up and close out on program exit a bit more neatly
  • untangled some messy value-change radio button code in the shortcut edit panel
  • updated the way page status text propagates up from the thumbnail grid to the main gui to Qt signals instead of the old inefficient pubsub
  • all UI file hash clipboard copying code is now unified and improved
  • added a new subscription file publish debug test to help->debug->gui
  • refactored some client specific time delta rendering code out of core to client
  • misc event cleanup code
  • misc code style cleanup

• version 401

  • subscriptions

  • as subs can now load more flexibly, previously hardcoded waits are now eliminated

  • - the subscriptions manager now only waits three seconds after initial session load to boot (previously 15)
  • - the subscriptions manager now wakes as soon as the subscriptions dialog is ok'd or cancelled
  • - a timing calculation that would delay the work of a sub up to five or fifteen minutes if more queries would come due for sync in that time window (in order previously to batch to reduce read/write) is now eliminated--subs will now start as soon as any query is due. if you were ever confused why a query that seemed due did not boot after dialog ok or other wake-up event, this _should_ no longer happen
  • re-added the import/export/duplicate buttons to manage subs. export and dupe may need to do db work for a couple of seconds and will have a yes/no confirmation on larger jobs
  • the import button on manage subs accepts and converts the old 'legacy' subscription object format, including a copy/paste of the objects backed up to disk in the v400 update
  • fixed an issue where creating a subscription query and then deleting it in the same manage subs dialog session would result in surplus data being written to the db (which the next dialog launch would note and clear out)
  • an unusual error with pre-run domain checking, exposed by the new subscription code and e621 subs, where the gallery url has also recently changed, is now fixed

  • issue tracker

  • the Github issue tracker (https://github.com/hydrusnetwork/hydrus/issues) is turned on again! it is now run by a team of volunteer users. the idea is going to be to try to merge duplicate feature suggestions with the proper platform and put some discussion and cognition and prioritisation into idea development before it gets to my desk, so I can be more focused and productive and so 95% of feature suggestions do not simply get banished to the shadow realm of the back of my todo
  • this is mostly intended for wishlist and other suggestions, as the tsunami was just getting too much for me to handle, but we'll see how it goes for things like bug reports as well. I'll still take any sort of report through my normal channels, if you are uncomfortable with github, or if you wish for me to forward an item to the issue tracker anonymously
  • the website, help documents, and hydrus help menu links have been updated regarding the issue tracker

  • the rest

  • improved how the database 'update default downloader objects' job works, ensuring that new defaults are better at simply take the place of existing objects and do not break/reset existing url class to parser links
  • tightened up how automatic url class to parser linking works, eliminating some surplus and potentially bad data related to api links. furthermore, whenever the links between url classes and parsers update, existing surplus data, which may creep in when api links change, is now cleaned from the data structure
  • rolling out updated e621 url class and parser to deal with their alternate gallery url format
  • rolling out an updated derpibooru parser that will link to the new api class correctly
  • thanks to a user's submission, rolling out updated versions of the new default nitter parsers that pull creator:username tags
  • before every subprocess launch, and when waiting for all subprocess communication (e.g. to ffmpeg), now tests regularly for program shutdown. if an unusual situation develops where a subscription is doing a file import job while the OS is shutting down, and that system shut down would hang or is hanging on a 'ffmpeg can't be launched now' dialog, the hydrus client should now notice this and bomb out, rather than going for that never-running ffmpeg. this may not fix all instances of this issue, and further feedback on the client not closing down cleanly with the OS is welcome.
  • when adding a new path to the 'migrate database' panel, any symbolic links will be converted to canonical equivalents
  • added some location checks and appropriate errors when the database is doing file storage rebalancing
  • fixed an issue uploading swfs, video, or audio to the server when it is launched from a frozen executable build
  • misc code cleanup

• version 400

  • subscription data overhaul

  • the formerly monolithic subscription object is finally broken up into smaller pieces, reducing work and load lag and total db read/write for all actions
  • subscriptions work the same as before, no user input is required. they just work better now™
  • depending on the size and number of your subscriptions, the db update may take a minute or two this week. a backup of your old subscription objects will be created in your db directory, under a new 'legacy_subscriptions_backup' subdirectory
  • the manage subscriptions dialog should now open within a second (assuming subs are not currently running). it should save just as fast, only with a little lag if you decide to make significant changes or go into many queries' logs, which are now fetched on demand inside the dialog
  • when subscriptions run, they similarly only have to load the query they are currently working on. boot lag is now almost nothing, and total drive read/write data for a typical sub run is massively reduced
  • the 'total files in a sub' limits no longer apply. you can have a sub with a thousand queries and half a million urls if you like
  • basic subscription data is now held in memory at all times, opening up future fast access such as client api and general UI editing of subs. more work will happen here in coming weeks
  • if due to hard drive fault or other unusual situations some subscription file/gallery log data is missing from the db, a running sub will note this, pause the sub, and provide a popup error for the user. the manage subscription dialog will correct it on launch by resetting the affected queries with new empty data
  • similarly, if you launch the manage subs dialog and there is orphaned file/gallery log data in the db, this will be noticed, with the surplus data then backed up to the database directory and deleted from the database proper
  • subscription queries can now handle domain and bandwidth tests for downloaders that host files/posts on a different domain to the gallery search step
  • if subs are running when manage subs is booted, long delays while waiting for them to pause are less likely
  • some subscription 'should run?' tests are improved for odd situations such as subs that have no queries or all DEAD queries
  • improved some error handling in merge/separate code
  • the 'show/copy quality info' buttons now work off the main thread, disabling the sub edit dialog while they work
  • updated a little of the subs help

  • boring actual code changes for subs

  • wrote a query log container object to store bulky file and gallery log info
  • wrote a query header object to store options and cache log summary info
  • wrote a file cache status object to summarise important info so check timings and similar can be decided upon without needing to load a log
  • the new cache is now used across the program for all file import summary presentation
  • wrote a new subscription object to hold the new query headers and load logs as needed
  • updated subscription management to deal with the new subscription objects. it now also keeps them in memory all the time
  • wrote a fail-safe update from the old subscription objects to the new, which also saves a backup to disk, just in case of unforeseen problems in the near future
  • updated the subscription ui code to deal with all the new objects
  • updated the subscription ui to deal with asynchronous log fetching as needed
  • cleaned up some file import status code
  • moved old subscription code to a new legacy file
  • refactored subscription ui code to a new file
  • refactored and improved sub sync code
  • misc subscription cleanup
  • misc subscription ui cleanup
  • added type hints to multiple subscription locations
  • improved how missing serialisable object errors are handled at the db level

  • client api

  • the client api now delivers 'is_inbox', 'is_local', 'is_trashed' for 'GET /get_files/file_metadata'
  • the client api's Access-Control-Allow-Headers CORS header is now '*', allowing all
  • client api version is now 12

  • downloaders

  • twitter retired their old api on the 1st of June, and there is unfortunately no good hydrus solution for the new one. however thanks to a user's efforts, a nice new parser for nitter, a twitter wrapper, is added in today's update. please play with it--it has three downloaders, one for a user's media, one for retweets, and one for both together--and adjust your twitter subscriptions to use the new downloader as needed. the twitter downloader is no longer included for new hydrus users
  • thanks to a user's submission, fixed the md5 hash fetching for default danbooru parsers
  • derpibooru gallery searching _should_ be fixed to use their current api

  • the rest

  • when the client exits or gets a 'modal' maintenance popup window, all currently playing media windows will now pause
  • regrettably, due to some content merging issues that are too complicated to improve at the moment, the dupe filter will no longer show the files of processed pairs in the duplicate filter more than once per batch. you won't get a series of AB, AC, AD any more. this will return in future
  • the weird bug where double-clicking the topmost recent tags suggestion would actually remove the top two items _should_ be fixed. general selection-setting on this column should also be improved
  • middle-clicking on a parent tag in a 'write' autocomplete dropdown no longer launches a page with that invalid parent 'label' tag included--it just does the base tag. the same is true of label tags (such as 'loading...') and namespace tags
  • when changing 'expand parents on autocomplete' in the cog button on manage tags, the respective autocomplete now changes whether it displays parents
  • this is slightly complicated: a tag 'write' context (like manage tags) now presents its autocomplete tags (filtering, siblings, parents) according to the tag service of the parent panel, not the current tag service of the autocomplete. so, if you are on 'my tags' panel and switch to 'all known tags' for the a/c, you will no longer get 'all known tags' siblings and parents and so on presented if 'my tags' is not set to take them. this was sometimes causing confusion when a list showed a parent but the underlying panel did not add it on tag entry
  • to reduce blacklist confusion, when you launch the edit blacklist dialog from an edit tag import options panel, now only the 'blacklist' tab shows, the summary text is blacklist-specific, and the top intro message is improved. a separate 'whitelist' filter will be added in the near future to allow downloading of files only if they have certain tags
  • 'hard-replace siblings and parents' in _manage tags_ should now correctly remove bad siblings when they are currently pending
  • network->downloaders->manage downloader and url display now has a checkbox to make the media viewer top-right hover show unmatched urls
  • the '... elide page tab names' option now applies instantly on options dialog ok to all pages
  • added 'copy_bmp_or_file_if_not_bmpable' shortcut command to media set. it tries copy_bmp first, then copy_file if not a static image
  • fixed some edit tag filter layout to stop long intro messages making it super wide
  • fixed an issue where tag filters could accept non-whitespace-stripped entries and entries with uppercase characters
  • fixed a display typo where the 'clear orphan files' maintenance job, when set to delete orphans, was accidentally reporting (total number of thumbnails)/(number of files to delete) text in the file delete step instead of the correct (num_done/num_to_do)
  • clarified the 'reset repository' commands in review services
  • when launching an external program, the child process's environment's PATH is reset to what it was at hydrus boot (removing hydrus base dir)
  • when launching an external program from the frozen build, if some Qt/SSL specific PATH variables have been set to hydrus subdirectories by pyinstaller or otherwise, they are now removed. (this hopefully fixes issues launching some Qt programs as external file launchers)
  • added a separate requirements.txt for python 3.8, which can't handle PySide2 5.13.0
  • updated help->about to deal better with missing mpv
  • updated windows mpv to 2020-05-31 build, api version is now 1.108
  • updated windows sqlite to 3.32.2

• version 399

  • improvements

  • the media viewer and thumbnail _right-click->manage_ menus now have a _viewing stats->clear_ action, which does a straight-up delete of all viewing stats record for the selected files. 'edit' will be added to this menu in future
  • extended the tag autocomplete options with a checkbox to allow 'namespace:' to match all tags, without the explicit asterisk
  • tag autocomplete options now permit namespace searches if the 'search namespaces into full tags' option is set
  • the tag autocomplete options panel now disables and checks the namespace checkboxes when one option overrules another
  • cleaned up some tag search logic to recognise and deal with 'namespace:' as a query
  • added some more unit tests for tag autocomplete options
  • the html and json parsing formulae now support negative indexing, to select the nth last item from a list
  • extended the '1 -> "1st"' ordinal string conversion code to deal with negative indices
  • the 'hide tag' taglist menu actions are now wrapped in yes/no dialogs
  • reduced the activation-to-click-accept time that the shortcuts handler uses to ignore activating clicks from 100ms to 17ms
  • clicking the media viewer's top hover window's zoom buttons now forces the 'media viewer center' zoom centerpoint, so if you have the mouse centerpoint set, it won't zoom around the button where you are clicking!
  • added a simple 8chan.moe watcher to the defaults, all users will get it on update
  • the default bandwidth rules for download pages, subs, and watchers are now more liberal. only new users will get these. various improvements to db and ui update pipeline mean the enforced breaks are less needed
  • when a manage tags dialog moves to another media, if it has a 'recent tags' suggestion list with a selection, the selection now resets to the top item in the list
  • the mpv player now tracks when a video is fully loaded and only reports seek bar info and allows seeks when this is so (this should fix some seekbar errors on broken/slow-loading vids)
  • added 'undelete_file' to media shortcut commands
  • file delete and undelete are no longer hardcoded in the media viewer and media thumbnail grid. these actions are now handled entirely in the media shortcut set, and added to all clients by default (this defaults to (shift +) delete key, and also backspace on macos, so likely no changes)
  • ctrl+mouse wheel is no longer hardcoded to zoom in the media browser. these actions are now handled entirely in the 'all' media viewer shortcut set (this defaults to ctrl+wheel or +/-, so likely no changes)
  • deleted some old shortcut processing code
  • tightened up some update timers to better halt work while the client is minimised to system tray. this _may_ improve some users' restore hanging issues
  • as Qt is happier than wx about making pages on a non-visible client, subscriptions and various url import operations are now permitted to create pages while the client is minimised to taskbar or system tray. if this applies to your situation, please let me know how you get on here, as this may relieve some restore hanging as the pending new-file jobs are no longer queued up

  • fixes

  • clicks on hover window greyspace should no longer propagate up to the media viewer. this was causing weird archive/delete filter actions
  • mouse scroll on hover window taglist should no longer propagate up to the media viewer when the taglist has no more to scroll in that direction
  • fixed an issue that meant preview windows were initialising about twenty pixels too short for the first page loaded in a session, and also pages created within nested page of pages. also cleaned up some logic for unusual situations like hidden preview windows. one more cycle of closing and reopening the client will fix the option value here
  • cleaned and unified some page sash setting code, also improving the 'hide preview window' option reliability for advanced actions
  • fixed a bug that meant file viewtime was still being recorded on the duplicate filter when the special exception option was off
  • reduced some file viewtime manager overhead
  • fixed an issue with database repair code when local_tags_cache is missing
  • fixed an issue updating a very old db not recognising that local_tags_cache does not yet exist for proper reason and then trying to repair it before update code runs
  • fixed the annoying issue introduced in the recent string match overhaul where a 'fixed character' string match edit panel would not want to ok if the (now hidden) example string input did not have the same fixed char data. it now validates no matter what is in the hidden input
  • potentially important parsing fix: JSON parsing, when set to get strings, no longer converts a 'null' value to 'None'
  • the JSON parsing formula now allows you to select the nth indexed item of an Object (a JSON key->value dictionary). due to technical limitations, it alphabetises the keys, not selecting them as-is in the JSON itself
  • images that do not load in PIL no longer cause mime exceptions if they are run through the decompression bomb check

  • misc

  • boosted the values of the decompression bomb check anyway, to reduce false positives. it generally now has a problem with images with a bmp > 1GB memory
  • by default, new file import options now start with decompression bombs allowed. this option is being reduced to a stopgap for users with less memory
  • 'MimeException' is renamed to 'UnsupportedFileException'
  • added 'DamagedOrUnusualFileException' to handle normally supported files that cannot be parsed or loaded
  • 'SizeException' is split into 'TagSizeException' and 'FileSizeException'
  • improved some file exception inheritance
  • removed the 'experimental' label from sub-gallery page url type in parsing system
  • updated some advanced help regarding bad files
  • misc help updates
  • updated cloudscraper to 1.2.40

• version 398

  • new tag search options

  • there are several new options for tag autocomplete under the newly renamed _services->tag display and search_

  • for 'manage tags'-style 'write' autocompletes, you can now set which file service and tag service each tag service page's autocomplete starts with (e.g. some users have wanted to say 'start my "my tags" service looking at "all known files" and "ptr"' to get more suggestions for "my tags" typing). the default is 'all known files' and the same tag service
  • the old blanket 'show "all known files" in write autocompletes' option under _options->tags_ is removed

  • you now can enable the following potentially very slow and expensive searches on a per-tag-domain basis

  • - you can permit namespace-autocompleting searches, so 'ser' also matches 'ser*:*', i.e. 'series:metroid' and every other series tag
  • - you can permit 'namespace:*', fetching all tags for a namespace
  • - you can permit '*', fetching all tags （╬ಠ益ಠ)
  • '*' and 'namespace:*' wildcard searches are now significantly faster on smaller specific tag domains (i.e. not "all known tags")
  • short explicit wildcard searches like "s*" now fire off that actual search, regardless of the 'exact match' character threshold
  • queries in the form "*:xxx" are now replaced with "xxx" in logic and display
  • improved the reliability of various search text definition logic to account for wildcard situations properly when doing quick-enter tag broadcast and so on
  • fixed up autocomplete db search code for wildcard namespaces with "*" subtags
  • simplified some autocomplete database search code

  • string processing

  • the new string processor is now live. all parsing formulae now use a string processor instead of the string match/transformer pair, with existing matches and transformers that do work being integrated into the new processor
  • thus, all formulae parsing now supports the new string splitter object, which allows you to split '1,2,3' into ['1','2','3']
  • all formulae panels now have the combined 'string processing' button, which launches a new edit panel and will grow in height to list all current processing steps
  • the stringmatch panel now hides its controls when they are not relevent to the current match type. also, setting fixed match type (or, typically, mouse-scrolling past it), no longer resets min/max/example fields)
  • the string conversion step edit panel now clearly separates the controls vs the test results
  • improved button and summary labelling for string tools across the program
  • some differences in labelling between string 'conversion' and 'transformation' are unified to 'conversion' across the program
  • moved the test data used in parsing edit panels to its own object, and updated some of the handling to support passing up of multiple example texts
  • the separation formula of a subsidiary page parser now loads with current test data
  • the string processing panel loads with the current test data, and passes the first example string of the appropriate processing step to its sub-panels. this will be expanded in future to multiple example testing for each panel, and subsequently for note parsing, multiline testing
  • added safety code and unit tests to test string processing for hex/base64 bytes outcomes. as a reminder, I expect to eliminate the bytes issue in future and just eat hashes as hex
  • cleaned up a variety of string processing code
  • misc improvements to string processing controls

  • the rest

  • double-clicking a page tab now opens up the rename dialog
  • system:time imported now has quick buttons for 'since 1/7/30 days ago'
  • all hydrus downloaders now accept percent-encoded characters in the query field, so if you are on a site that has tags with spaces, you can now enter a query like "simple%20background red%20hair" to get the input you want. you can also generally now paste encoded queries from your address bar into hydrus and they should work, with the only proviso being "%25", which is "%", when all bets are off
  • duplicates shut down work (both tree rebalancing and dupe searching) now quickly obeys the 'cancel shutdown work' splash button
  • fixed a signal cleanup bug that meant some media windows in the preview viewer were hanging on to and multiplying a 'launch media' signal and a shortcut handler, which meant double-clicking on the preview viewer successively on a page would result in multiple media window launches
  • fixed an issue opening the manage parsers dialog for users with certain unusual parsers
  • fixed the 'hide the preview window' setting for the new page layout method
  • updated the default gelbooru gallery page parser to fix gelb gallery parsing
  • updated the newgrounds parser to the latest on the github. it should support static image art now
  • if automatic vacuum is disabled in the client, forced vacuum is no longer prohibited
  • updated cloudscraper for all builds to 1.2.38

  • boring code cleanup

  • all final mouse event processing hackey is removed from the media viewers, and the shortcut system is now fully responsible. left click (now with no or any modifier) is still hardcoded to do drag but does not interfere with other mapped left-click actions
  • the duplicates filter no longer hardcodes mouse wheel to navigate--whatever is set for the normal browser, it now obeys
  • cleaned up some mouse move tracking code
  • clicking to focus an unfocused media viewer window will now not trigger the associated click action, so you can now click on archive/delete filters without moving on!
  • the red/green on/off buttons on the autocomplete dropdown are updated from the old wx pubsub to Qt signalling
  • updated wx hacks to proper Qt event processing for splash window, mouse move events in the media viewer and the animation scanbar
  • cleaned up how some event filtering and other processing propagates in the media viewer
  • deleted some old unused mouse show/hide media viewer code
  • did some more python imports cleanup
  • cleaned up some unit test selection code
  • refactored the media code to a new directory module
  • refactored the media result and media result cache code to their own files
  • refactored some qt colour functions from core to gui module
  • misc code cleanup

• version 397

  • regular changelog

  • added 'system:has/has no note with name xxx' to search for specific note names
  • in the normal system predicate list, the notes pred is now the generic 'system:notes' to launch a combined dialog for both num notes and named notes
  • favourite tag suggestions are now sorted in manage tags dialog according to the default tag sort
  • page names will now middle...elide when there are too many to fit into a row (and normally left/right buttons would be added). if the elided tabs still do not fit, the buttons will pop up as before. added a checkbox to options->gui pages to turn this text eliding off
  • pulled the 'page name' options on that panel into their own box and added some text regarding the 'my big row of import page tabs keeps scrolling weird' issue
  • when files are pixel duplicates, the filesize and age comparison statements will now have 0 score and thus be coloured neutral blue
  • the standard text entry dialog now always selects any default text it starts with, so you can now type to immediately overwrite. see how you like it and if there are some places where you think an exception should be made
  • updated the IPFS interface to work with the new IPFS 5.0. all api requests are now POST so it doesn't 405, and the User-Agent is overridden to one that IPFS will not 403 at, and I fixed a typo the new api is more strict about
  • a hack to get page splitters to lay out correctly on session load is rewritten from a hammer to a scalpel. pages now set their splitter positions on their first individual visible selection. this both reduces some minor ui lag on session/page load and improves splitter positions for clients that open minimised to the system tray
  • a long-time odd issue where loaded sessions would initially select the top-left-most non-page of pages is fixed. now the bottom-left-most page of any kind is selected
  • fixed tag autocomplete selecting the bottom-most pre-loading result. it now correctly selects at the top
  • fixed an issue setting certain values (typically loading a default) to a tag import options panel
  • the client is now more aggressive about clearing subscriptions from memory when they are finished running
  • in windows, the main method that copies files now checks for modified time of the source file. if it is before 1980-01-01 UTC, it does not copy the file metadata, as some Windows has trouble with this lmaoooo
  • cleaned up how some thumbnail 'current focus' media determination code works. should have fixed some weird errors when hitting certain shortcuts on collections
  • cleaned up basic list/sort code across the program
  • the 'queue' and add/edit/delete listboxes now emit change signals when new items are added or imported
  • pyparsing, a helper for cloudscraper, is now correctly bundled in the built releases. a new line in help->about displays this
  • help->about now lists cloudscraper version
  • updated the discord link to the new https://discord.gg/wPHPCUZ

  • upcoming string processing changes for advanced users

  • I extended string parsing code this week, but I am not yet ready to turn it on. when it does come on, it will change all formulae from the fixed string match/converter pair a combined general string processing 'script' of n steps
  • wrote a new 'string splitter' object that takes one strings and splits it into up to n strings based on a separator phrase (such as ' ,')
  • wrote an edit panel for string splitters
  • wrote a new 'string processor' object that holds n ordered string match/converter/splitter objects and filters/converts/splits x strings into y strings based on those steps
  • wrote an edit panel for string processors. it has a notebook that live updates with test results for each step on every update
  • wrote unit tests for string match
  • wrote unit tests for string converter
  • wrote unit tests for string splitter
  • wrote unit tests for string processor
  • refactored string conversion edit panels to their own file
  • refactored string conversion controls to their own file
  • misc string processing cleanup and labelling improvements

  • technical url parsing stuff

  • urls are now stripped of leading and trailing whitespace during normalisation, just in case a paste contains some extra whitespace. previously, it would sometimes throw a 'doesn't start with http' error
  • the hydrus url normalisation process now normalises the hostname according to the NKFC unicode format, meaning unusual characters like ？and e◌́ are now replaced with their normalised visual equivalent ? and é, and hence these urls will no longer throw errors when they are added
  • if '?' or '#' end up in a hostname (which are invalid characters), it is now converted to _, just to stop complete parse mangling when weird urls are submitted. this character replacement may become more sophisticated in future
  • the hydrus downloader should now support search terms that include '#'
  • download query parameters that contain '%23' ('#', encoded) are now not unquoted in url normalisation

• version 396

  • notes

  • the file notes system is more mature. files now store multiple named notes
  • the edit notes ui is now a tabbed window with add/edit_name/delete buttons
  • media results now load with their notes, so note access is instant
  • thumbnails now show a notes icon when they have notes
  • the media viewer top-right area shows a notes icon when the current file has notes
  • clicking the media viewer top-right notes icon opens edit notes
  • the edit notes menu entry now lists the number of current notes if there are notes
  • added a 'system:number of notes' predicate. it has easy 'has/no notes' buttons for quick filtering
  • the file notes database table will be updated on update, it shouldn't take long. existing notes will get the default 'notes' name
  • duplicate notes now share the same storage space in the database
  • in prep for a future search expansion, notes are now cached in the database for fast text search
  • in prep for note parsing, wrote a 'note import options' object. it doesn't do anything in the program yet, but it supports multiple note conflict resolutions, note extension detection, and global and specific note renaming
  • wrote unit tests for the new note import options

  • some tag search stuff

  • hydrus now maintains an internal mapping of direct 'searchable' versions of tags to the tags themselves, which allows it to now do fast exact-match (short search) and complicated wildcard lookups of tags with unusual characters. 'f' and '/f/' will now return '/f/' and 'board:/f/' quickly, 'board:f' and 'board:/f/' will return 'board:/f/' quickly, and 'te*a*' will correctly return 'test-tag'
  • it will take a few minutes to regenerate this new cache on update
  • complex wildcards like 's*m*' are now treated the same as simple ones like 'sam*' and should match unusual subtag characters in all cases
  • wildcard tag file search predicates are now plugged into the new cache, so the search preds '/f/*', 'board:/f/', 'board:/f/ast;', 'b*d:/f/' and 'b*d:/f/*' now all match files with 'board:/f/', as do wildcards that include replacement characters, so the same should be true above for 'f' instead of /f/' in all cases
  • new wildcard search preds do not collapse their characters for their presentation string, so 'date:2*-01-01' now renders like that, not 'date:2* 01 01'
  • wildcard file search predicates are now faster for simple (just an asterisk on the end) subtag wildcards
  • the fts search cache is moved from 'master' to 'caches' db this week, it will take a few moments on update
  • the 'repopulate tag search cache' db regen job now repopulates the fts cache, the new 'searchable' cache, and the integer tag cache
  • the database repair code now checks for the fts cache and new 'searchable' cache on boot and, if they are missing, warns the user and creates empty tables

  • improvements

  • fixed the unsorted tags in tag suggestion boxes
  • clicking the inbox icon in the top-right hover window now archives the file
  • system:dimensions now has quick buttons for 16:9, 9:16, 4:3, 1:1, 1080p, 720p, and 4k
  • system:known url searches are now better about fetching www and non-www urls for the domain or url class
  • the edit shortcut sets panel now has nicer english names for reserved shortcut sets, and also sorts them in a more logical way
  • you no longer have to be in advanced mode to copy file hashes from thumbnails
  • users in advanced mode can copy the internal file_id of files from the thumbnail/viewer copy menus (this is most useful for the client api)
  • system num_frames, num_words, and num_notes now display alternate 'has/no xxx' labels when they search for =0 or >0
  • you can now search for 0 with system:num_frames

  • fixes

  • users who could restore from system tray using the menu but had trouble with clicking _should_ now have better luck with clicking
  • fixed some instances where fps could be calculated as 0, which would lead to other problems down the line. now a missing or 0 fps is remapped to 1
  • fixed system:framerate for '<' queries
  • the status bar cells now get expanded tooltips to describe what they do
  • fixed some media result caching code that could in rare cases cause an error in content update processing when the result disappeared from the cache during processing
  • the 'hard-replace siblings and parents' button on 'manage tags' now makes a submenu so its actions' long labels show better
  • fixed a handful of tables that were not starting sorted
  • a variety of credential parse and other server failures that were formerly returning 403 now properly return 400 and 409
  • in order to improve default 'open externally' behaviour on Linux/macOS, if the environment variable XDG_DATA_DIRS is not preserved through a hydrus build launch env, hydrus now sets a simple 'default' value for this before running xdg-open
  • if the client is booted from a windows shortcut to a built release, the program restart command is slightly more reliable

  • misc

  • cleaned up some db update error reporting code, it should now more reliably make an english-friendly popup text box before splurging technical info
  • refactored some media object code, cleaned some class definitions, and added typing hints
  • misc code cleanup
  • the 'getting started' help files now have anchor definitions, so their sections can now be #linked to
  • added several links in the 'getting started' help to the user-created video guides here: https://github.com/CuddleBear92/Hydrus-guides thank you for making these!
  • added a link to the help for the user-made 'other archiving software' guide here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/0-Alternative-Programs-and-Resources#software thank you for making this!
  • fixed link to AUR package in the help
  • updated cloudscraper in all builds to 1.2.36
  • updated windows mpv to a significantly newer dll, it now reports api version 1.108
  • included libgpg-error.so.0 in Linux build, which will improve some Linux situations (more reports from Ubuntu 20.04 or others about missing/conflicting .so files are welcome)

• version 395

  • some more suggested tags fixes/qol

  • favourite tags now correctly refreshes on new media
  • the tag suggestion lists in manage tags now discard current and pending tags that _all_ the current media already have, and all tag suggestion lists update this filter any time the media gets a tag content update! they _should_ update live now
  • all tag and predicate taglists now try to move the selection to a 'nice' neigbour when a keyboard enter activation results in the current selection being removed (e.g. as in these tag suggestion lists). the nice selection should be the tag after, before, or at the top of the list, and should make it nicer to keep navigating the list and add tags with your keyboard
  • all tag and predicate taglists now try to preserve selection on simple clear-and-set data refreshes

  • deleted tags overwrite update

  • due to an unfortunate oversight, until now tag parsing has not filtered out previously deleted tags from the tags it parses and sends to the local database
  • as the majority of downloaded files are parsed once per site per user and in a similar time window before manual editing ever occurs, and most non-tag-sibling-eligible bad tags are site specific or not parsed to begin with, and as these undesired tags were not broadcast up to the tag repository, this problem has not been very obvious and I believe has not affected most users too much. this is however a reason why some users who have more recently downloaded many older files are seeing smaller 'deleted mappings' counts on their ptr review panel (and some low quality tags in their db), as they have been re-adding previously deleted tags to their local store
  • this has been fixed. tag import options now load the pending importee file's metadata before tags are filtered and discard currently deleted tags from those to be added or pended. this applies to parsed tags, additional tags, and those tags added through special other means, such as from a parent gallery page.
  • if you do wish to allow parsed or additional tags to overwrite currently deleted tags for a particular job, the cog icons on the edit tag import options panel now allow you to permit overwrite for either
  • tags added via hard drive imports or the migrate tags tool still overwrite deleted tags as before
  • as this is a local-only problem, there is thankfully a retroactive fix for this issue for tag repository domains, involving a content reprocess run to re-apply deleted tags. I am not activating this automatically this week as this is a heavy job for the ptr and I need to study the true fallout of the problem more, but I may in future, likely as a smaller and more targeted maintenance job. advanced users can do it now under the ptr's review services panel
  • I regret missing this, and I am sorry for any inconvenience. I only discovered it through the serendipity of some users recently reporting unusual deleted counts and a personal item in my todo to check the reliability of deleted mapping filtering for local tag domains--turns out it never got added, and we never specifically noticed, fugg
  • there are now unit tests for the improved tag filtering pipeline and both of these new overwrite options

  • the rest

  • hydrus can now use several different zoom 'centerpoints' about which to expand and shrink a zooming file. this was previously hardcoded to the center of the media. under options->media, you can now set it to be the media window center (the new default, which feels much nicer after a pan), the mouse cursor, the old media center, or the media top-left corner
  • cleaned up the related zoom positioning code, and removed the jarring old re-centering off-screen rescue hack when zooming out to canvas zoom
  • added a warning about big zooms to the media options page
  • fixed tag autocomplete filtering in python 3.7 so 'character:aran' matches 'character:samus aran' again
  • when the hover windows on a media viewer have focus, they _should_ now pass up all options->shortcuts shortcuts to the media viewer
  • mouse back/forward buttons _should_ now be supported in the shortcuts system, as much as your OS allows them to work like regular clicks
  • fixed a rare crash with the 'clear trash' button
  • the client will now not re-analyze tables that have been previously scanned with at least 100k rows in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
  • the client will now not vacuum database files greater than 1GB in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
  • the new 'cannot vacuum because xxxx' log entry is now only ever printed once per boot. however due to the above change, it likely won't appear in the normal maintenance cycle anyway now
  • cleaned up some vacuum code
  • reworked the panel system to better test data validity vs 'woah, you sure you want to do this?' tests and generally cleaned and simplified the canok/cancancel/isvalid testing logic for all panels. panels like manage siblings will now not produce two message boxes if you try to ok them on an uncommited pair and then back out of the ok
  • refactored the top level window code and improved scrollable panel code typing
  • more standalone gui function code refactoring
  • fixed a click-selection-test bug when clicking on certain whitespace in certain predicate lists
  • the text of the cloudflare-specific error when encountering a captcha page is improved
  • cleaned up some tag list menu copy and select code, both the menu labels and the copy action, for unusual tags. the 'copyable tags' fetching code is now flexible and unified for menu and action
  • cleaned up the taglist sibling copy code, eliminating the chance of dupes
  • fixed a _little_ of the wording on the discard/exclude tag list menu labels for negated predicates, it still feels a bit awkward and I will keep working here
  • cleaned up some old media metadata fetching code
  • misc import code typing
  • misc list/iterable typing improvements
  • added some misc media-tag tool code
  • unified the tag import options tag filtering pipeline somewhat to deal with the deleted overwrite situation
  • improved a debug ui test to no longer need window focus
  • misc help cleanup

• version 394

  • autocomplete cleanup

  • the text you type into tag autocomplete is now parsed in a unified object. all the variants of empty text, invalid text, valid text, namespace text, and wildcard text are all tested and fetched in one simple location with better code
  • autocomplete results caching is now a unified object that tracks and filters results in one location. wildcard searches are now never cached by accident, and switching from tag cache to system predicate cache and to non-initialised cache is instant and more reliable
  • when an autocomplete, either in a search page or a context that manages tags, has results include multiple sibling variants of the typed text, they are now all elevated to the top of the list. the ideal is at the top, the entered text is next, and any known siblings follow
  • the search character 'collapse' that ensures quote marks and hyphens and other odd characters are unified across tags now applies uniformly to all non-complicated-wildcard search tags, with namespace not collapsed and subtag always collapsed
  • when entering an explicit wildcard search, both strict and autocomplete versions (whether they end with an asterisk) are now displayed
  • the way tag results are filtered is now more accurate for some unusual wildcards
  • it is now more difficult to slip cpu-killer search tags (weird asterisk combinations) through
  • the quick-broadcast that happens when the user hits enter before any results have started loading now uses the unified object and chooses a safer and more reliable broadcast value. the test whether to do the quick-broadcast is also more reliable, particularly in unusual situations where a recent search was cancelled or delayed. note that for many users, the cache and search tech is fast enough that this very rarely triggers
  • searching with a wildcard below the autocomplete threshold can no longer trigger a full search, nor an invalid exact-text search
  • namespace count merging is now unified across db tag fetches and media fetches
  • include current/pending buttons now filter down to media-based tag autocomplete counts
  • namespace tag autocomplete queries will no longer show up some unusual siblings below the 'anything' tag
  • deleted a whole bunch of old a/c and caching code
  • added comprehensive unit tests for the new parsed autocomplete text object
  • added comprehensive unit tests for the new predicate results cache object

  • the rest

  • fixed a stupid typo bug in the new domain checking code that was stopping subscriptions with incomplete file queues from starting. I apologise for this
  • network error responses 502 (Bad Gateway) and 503 (Service Unavailable) are now treated as a retryable. the 503 is assuming it is not a CF challenge page. if they fail all retries, they are considered a network infrastructure error
  • all other misc 5xx http responses are now treated as instant network infrastructure errors and will be logged in the new domain health tracker
  • the exit splash screen now opens a bit earlier, so you now shouldn't have any momentary uncertainty where no windows are open
  • clients that start minimised to system tray _should_ be better about restoring splitter positions on first show
  • the various 'management panels', the panels on the left of main gui pages, now have smaller minimum width where available. the gallery and watcher panels are still the widest, which is a limitation of the current list tech. when it gets better column sizing code and selection memory, this will improve
  • fixed an issue loading gifs with some OpenCV versions
  • brushed up some running from source help
  • deleted the Py2To3 script that attempts to detect a legacy python 2 install
  • improved all the gui files' import order
  • cleaned up and refactored some subscription code
  • added a bunch of type hints to edit panel code
  • misc code cleanup

  • environment updates

  • did second step of hydrus project structure improvement--now the project is split into subdirectories for core/client/server/misc and some client subdirs. work here will continue
  • linux build gets some new libraries, cv is up to 4.2.0
  • it isn't important, but hydrus is now built in python rather than directly from command line. my build scripts now include cloudscraper and the new hydrus source code tree in the build as they are, rather than hardcoded copying

• version 393

  • cloudflare and network

  • the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, but only a handful of the more complicated captcha ones
  • if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information
  • the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays
  • basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage
  • subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth
  • serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors
  • I expect this system will need more tuning
  • the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file

  • the rest

  • the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems
  • similarly, all the windows libraries are now their latest versions. opencv is now 4.2
  • greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain
  • thanks to some excellent work by a user, the Deviant Art downloader gets another pass--it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok
  • tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules
  • the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check
  • added some unit tests to test the new tag import options blacklist namespace rule
  • when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable
  • the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on
  • the system tray now has a tooltip with the main hydrus title and pause statuses
  • the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own
  • added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets
  • added a semi-hacky duplicate button to the 'manage http headers' dialog
  • the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog
  • a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds
  • the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain
  • fixed a bug when petitioning tags through the client api, with or without reasons
  • fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open
  • the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex
  • added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out
  • updated running from source help with cloudscraper, a new pip masterline, and some windows venv info
  • the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import >>'. it also gets a tooltip
  • handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown

• version 392

  • db-level tag sibling cache

  • the hydrus client db now maintains a fast cache of current+pending tag-to-ideal-tag sibling relationships. it works for specific services and 'all known tags'. this is a nice tool and the first step in having a proper hard-baked siblings mappings cache
  • the new sibling cache can be regenerated under _database->regenerate_. the 'autocomplete cache' entry under that menu is also renamed to the now more appropriate 'tag mappings cache'
  • the db repair system can regenerate this new cache if any part is missing on boot
  • the lookup that finds tag sibling matches for autocomplete uses this and is now faster, specific to the searched service, more accurate about status, and now includes pending siblings
  • wrote a new unified object to manage a collection of tag siblings, it is now in use at the db level
  • as I continue to develop this new fast tech, the old 'apply all sibs to all services' option, which was always buggy, may sometimes not apply in it. I will ultimately replace it with a fuller per-service choice system that will work quickly and properly and in the same unified way
  • fixed a bug where only one local tag service's siblings would be matched at the ui level when looking at 'all known tags'
  • fixed a bug in the file search code where searching for a tag that had an unnamespaced sibling going to it would result in searching for all possible namespaces of that sibling (e.g. searching for 'character:samus aran' when 'samus aran'->'character:samus aran' sibling existed would result in effectively 'anything:samus aran')
  • when tag services are deleted, they are now better about cleaning up their siblings and parents quickly
  • optimised some tag and hash id->value database cache population routines to improve performance for large queries (e.g. when fetching all the tag parents/siblings on boot). also these caches are now larger, 100k instead of 25k
  • all cache regen code now forces an immediate analyze of the new tables to speed up imminent access

  • the rest

  • updated the default e621 file page parser to get rating tags again (looks like their markup just changed again)
  • updated the default sankaku file page parser to get their recently redefined 'genre' tags
  • in edit subscriptions, the 'overwrite tag import options/check options' actions now initialise their dialogs with the current value for the first subscription, rather than the global program default
  • in the edit subscription panel, the checker options button is moved down to the file/tag import options
  • when not in advanced mode, the edit tag import options panel now has some red-text at the top to reinforce to new users that they should generally use the defaults
  • the tag import options blacklist now secondarily checks against all known siblings of the parsed tags, rather than just the 'collapsed' ideal siblings
  • subscriptions are now more aggressive about clearing out old urls from their file import caches--instead of clearing the 251st url after it has aged twice the death period, now they use just one DP. also, checkers with static checker timings will use five times that check period as DP if that is smaller. static checkers, or those that never die, will use a flat value of six months as DP if that is smaller
  • moved a bunch of the debug 'data actions' to a new 'memory actions' menu
  • significantly reduced how often the system tray regenerates its menu, which seems to improve stability
  • fixed an issue where guis that were maximised before a minimise were restoring from a system tray icon click to normal view
  • double-clicking the system tray when the ui is hidden should no longer do a fast show/hide
  • fixed an issue where if the gui was minimised, the main animation timer would not run for other windows (e.g. a separate media viewer)
  • improved ui shown/hidden tracking logic for the new system tray icon for different OSes
  • fixed the 'refresh_page_of_pages_pages' shortcut action, which had faulty old wx code in it
  • fixed a wx->Qt bug where modal popups that cannot be cancelled, and thus pop up a 'sorry, you can't dismiss this' text when you try to close them, were nonetheless still closing afterwards
  • the hydrus client and server now attempt to listen their servers on both IPv4 and IPv6, failing gracefully if IPv6 is not available
  • the 'is this a localhost request?' check now understands IPv6 localhost (::1 or ::ffff:127.0.0.1)
  • may have solved a 100% cpu repaint issue with the a/c dropdown in some qt environments
  • added info to installing help about Windows N and clean installs
  • misc media viewer wx->Qt code cleanup
  • misc code cleanup

  • experimental hellzone, be wary ye scabs

  • added an experimental 'sub-gallery url' url content type to the parsing and downloading system. this url is queued into the gallery log even if the primary gallery page found no file/post urls, and is meant for galleries that link to galleries. not yet ready for primetime, but feedback would be appreciated
  • added an experimental ui-hang relief mode, activated under _help->debug->data actions->db ui-hang relief mode_, which _should_ stop the ui hanging in unusual long-time ui-synchronous db jobs. it may cause other problems, so it is default off. it also prints begin/end statements to log for additional info. users who experience ui hang due to db job processing time are invited to play with this mode and report back results

• version 391

  • system tray icon

  • hydrus now can now make a system tray icon for those OSes that support it. it can be buggy/crashy under non-Windows, where it gets some warning labels
  • under the new options->system tray page, you can set whether to show the system tray all the time, minimise the main gui to system tray, close-button the main gui to system tray, and start the program minimised to the system tray
  • right-clicking the icon brings up a menu to show/hide the ui, pause/unpause network traffic or subscriptions, and to exit hydrus
  • the main file menu now has an option to minimise to system tray
  • double-clicking or middle-clicking the icon will show/hide the whole hydrus ui as long as there are no dialogs open
  • clicking it will restore the main gui from minimise or raise it to the front
  • on an ui hide, the current preview window will be blanked and media viewers will be paused, so any ongoing noise/cpu from them should stop
  • a new 'global' shortcut 'hide_to_system_tray' is now available
  • starting the client minimised may have some layout issues on first show--I particularly had to fix splitter layouts--please report any more you discover

  • framerate and num frames

  • system:framerate search added to system:duration panel. precise framerate is tricky with current hydrus info, so it searches +/- 5% of a given value
  • system:number of frames added to system:duration panel
  • sort by number of frames added
  • duration/framerate/num frames sort moved to their own 'duration' submenu
  • framerate added to generic media metadata summary string (which appears in status bar and media viewer, etc...). precise framerate is tricky with current hydrus info, so it is rounded to the nearest integer

  • the rest

  • rolling in new danbooru file page parsers that should fix file downloads, thank you to a user for the submission
  • rolling in a e621 login script, thank you to a user for the submission
  • gave tag autocomplete results fetch code a pass, cleaning up several instances of incorrect or inefficient timing and caching logic and I believe fixing the issue where system preds would sometimes not be loaded after entering a tag
  • improved reliability of autocomplete dropdown hiding on background pages (some edge cases where these could still hang around _should_ be fixed)
  • improved 'hide' tests in several parts of the program related to the new system tray icon, which should help some other cases--e.g. weird shutdowns now _shouldn't_ ever leave a bunch of floating popup messages
  • fixed a bug where pages set to open with all known files/tags domains, which is not supported, was incorrectly substituting tag domain with a file domain, which is even more not supported
  • cleaned up some sort code--I believe this has fixed the odd issue where a 'time imported' sort would not work on some pages (such as one loaded from a favourite search)
  • fixed the 'related' tag suggestion box not knowing about new pending tags added in a manage tags dialog open on a media viewer after next/previous media transitions while the dialog is open. also it and the file lookup's lists now clear when a new lookup starts
  • the tag suggestion boxes are now add-only and remove what you add as you add them! let me know if this feels nice or not!
  • the splash window now has a different 'booting/exiting' window title, if you would like to hook it with a window manager
  • went over all the 'prep url for display', 'filter urls', and 'normalise url' requests across the program to deal with invalid url (e.g. garbled text) better
  • you can now no longer add invalid urls via the client api associate_url call--you'll get 400 instead
  • cleaned some thumbnail selection and rendering code, particularly fixing some edge case 'where that media go?' issues where collect-by calls happen during thumbnail waterfalls and so on
  • cleaned up some page file domain setting code and misc page management code
  • improved accuracy of rendered image cache memory footprint calculations
  • fixed some Qt signal object definitions that were causing errors for some users who run from source

• version 390

  • fixed a bug that was causing potential duplicates to be sometimes re-added between media groups that were previously set as false positive/not related. I apologise for the inconvenience this bug has caused. if you were hit by this, please reset your potential duplicate pairs (hit the cog button on the dupes page) and re-search, and the bad pairs should not be re-added again
  • fixed an issue where tag autocomplete entry in the form 'namespace-blah:' was replacing the hyphen or other 'collapsable' character to space, which then was not searching correctly for the _anything_ namespace search
  • 'namespace:anything' searches now work when the namespace itself has a wildcard
  • fixed 'write' autocompletes not matching inputs with UPPERCASE letters
  • fixed adding tags that start with a colon (e.g. ":D") in 'write' autocompletes
  • it should now be impossible to enter some 'kill my cpu' queries into tag autocomplete, such as '[asterisk]:anything', even if accidentally entered through the fast-add system
  • the 'cancel search' stop button that appears after a search takes three seconds is back to being neatly embedded beside the tag autocomplete input box
  • hitting the cancel search button now clears the non-interactable 'Loading' thumbnail media page (with its misleading 'Loading...' statusbar) and returns you to an empty thumbnail page
  • loading a favourite search with non-immediate search no longer loads the 'loading...' page. it also saves that new non-immediate status to the page session more reliably
  • reworked my linux build environment (pyinstaller=3.5, virtualenv=16.1) so that you can launch the built exe using a symlink
  • rolling out a first version of a requirements.txt, any feedback would be appreciated
  • rolling out another version of the derpibooru file page parser that no longer duplicates namespaced tags as unnamespaced, thank you to the user who submitted this

  • boring stuff

  • moving old pubsub system to Qt signals

  • all the 'refresh query' calls that do changes to the current file search across the program
  • the current file and tag domain update calls for search pages
  • the clear/set file focus calls when launching and exiting the media viewer browser or archive/delete filter
  • the way thumbnails send the current focused media to the preview media window
  • the way widgets with shortcuts-based tooltips are notified to update those tooltips when shortcuts change
  • the way a thumbgrid sends the current tags to be displayed in the 'selection tags' list
  • the way a thumbgrid adds newly imported files' tags to the 'selection tags' list as they fade in
  • the way the 'searching/waiting' search button is flipped on and off by shortcut. btw what should be the correct name + label for this button? should it really be an icon?

  • cleanup

  • NOTE: the 'include' folder is renamed this week to 'hydrus'. if you have source patches, please update. as I further disentagle code in future, hydrus will ultimately move to typical nested folder/module structure
  • decoupled the shortcuts edit ui code from the controller and db, unified how shortcuts are edited, and eliminated db wait when booting shortcuts editing
  • decoupled the shortcuts manager from the controller, cleaned all the code, and moved to a nicer reference with proper typing hints
  • refactored the frame and media controls of clientguicanvas into separate files
  • renamed the hoverframes file to canvashoverframes and updated its classnames to 'canvas' rather than the old 'fullscreen'
  • fixed two wx->Qt typo artifacts in the login script edit ui
  • reduced some occasional idle memory bloat of clients that have large subscriptions
  • cleaned up how media-based taglists are appended with new media
  • removed some old booru object update code
  • some misc setmedia/clearmedia cleanup
  • misc search code cleanup
  • misc typing hints to clear up pylint confusion
  • misc tag autocomplete code cleanup
  • misc 'global' variable cleanup
  • misc gui code refactoring, cleanup and typing

• version 389

  • downloaders

  • the e621 file page parser is updated again, thanks to a user's contribution. this one gets md5 and file url more reliably, and also gets rating tag
  • added a 'e621 file page (old format)' url class to help match and search for files downloaded with the old format. please be aware there is no good solution to auto-convert old urls to a new format yet, so this connection does not (yet) solve the old/new comparison test
  • updated deviant art file post parser to use their json api. this should be more resilient to their current layout changes
  • the nijie.info login script appears no longer to function. as with exhentai last week, it has been removed to make it easier to log in with hydrus companion. please use hydrus companion if you would like to log into nijie.info
  • updated file lookup scripts for 'iqdb danbooru' and 'danbooru md5' thanks to a user's contribution

  • the rest

  • the way the mpv.conf works changes this week. it is now correctly fully portable, stored in the db directory beside the .db files. if this file does not exist, the 'default' as stored under the install_dir/static/mpv-conf folder will auto-populate it. if you have been using a non-default mpv conf, please re-set it one time after update, and you should be good
  • the code that loads mpv.conf is now more graceful on 'missing file' errors, which now means when both the db conf and the default conf are missing
  • hitting escape on a tag autocomplete input that has text will now clear that text! note that hitting escape on an _empty_ a/c input will still do 'lose focus' and then 'close dialog'
  • updated the slideshow logic so that if a media with duration has a shorter duration than the slideshow duration (e.g. a gif that lasts 0.5s on a 10s slideshow), the media will keep looping until the duration is up. media that has duration longer than the slideshow time will continue to play through once completely, delaying slideshow progression and then stopping promptly when it has finished
  • the string transformation system now allows 'url percent encoding' under the encode/decode type!
  • fixed the 'only add existing tags' filter in the tag import options, which was denying all the tested tags. it seems to have been hit by a typo in the last three months
  • the 'favourite searches' defaults now include an 'empty page' entry, which is a convenient way to simply clear a page. all users will also get this on update, feel free to delete if you don't like/need it
  • opening a new search page from a tag or an active search predicate ('open a new search page for...' or middle-click) now copies the file service (e.g. looking at trash) from the original page
  • opening a new search page in the 'all known files' file domain when the tag domain should be 'all known tags' (a currently unsupported combination) now coerces the tag domain to 'all local tags'
  • checkboxes should now appear again on the collect-by dropdown in Fusion (and hopefully any other) style
  • fixed an issue where entering 'namespace:*' explicitly would show the much less efficient wildcard search rather than the efficient 'anything' namespace search
  • fixed an issue where wildcard search could include multiple asterisks in a row
  • fixed an issue with page duplication where the main management object was not being duplicated properly until a session reload, meaning the two pages would sometimes share signals and changes
  • an old wx delayed hide/show performance hack is removed, making the floating autocomplete dropdown now update more smoothly to resize or move requests, such as when the main gui window is dragged
  • the program base installation directory is now calculated more accurately, both when running from source and the frozen build, and when launched using a symlink
  • install dir and db dir are now specified in the help->about window
  • the petition page content checkbox list now has a taller minimum height
  • improved error text reporting in hydrus service login failure, hydrus service delay reason-setting, and all 'cancelled' errors across the program
  • the review services panel now has elided... text. when unusually long errors propagate up to its status texts, it now won't suddenly jump to 2,000 pixels wide. full text appears in tooltips
  • code refactoring: the tag autocomplete input now now takes responsibility for the active predicate list above it
  • refactored some tag lists and added typing hints to improve how current page predicates are determined
  • did some prep work for tag filters supporting wildcards, but it isn't ready yet
  • cleaned up some wx->Qt data fetching code
  • misc code cleanup

• version 388

  • favourite searches

  • hydrus can now save, load, and edit favourite searches. this first system stores searches with a name and an optional folder name, and contains search predicates, file and tag domain, whether the search is live or not, and optionally sort-by and collect-by
  • this is program-wide and all accessed through the new 'star' icon menu button beside any 'read' tag autocomplete input on search pages, duplicate pages, export folder ui, and file maintenance selection
  • wrote a favourite searches manager
  • wrote a dialog to manage favourite searches
  • wrote a dialog to edit a single favourite search
  • wrote load and save search functionality
  • autocomplete dropdowns that have buttons beside them now stretch their floating dropdown windows across the button width also
  • cleaned a variety of search code, simplifying objects and responsibility
  • cleaned up some collect-by ui code
  • refactored sort and collect controls to better location
  • refactored search constants
  • numerous small search code fixes and cleanup
  • renamed clientguipredicates to clientguisearch

  • the rest

  • a note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion
  • the default e621 downloader is updated to their new system, thanks to a user's submission. if you log in to e621 with hydrus or the hydrus companion and discover some tags are now blacklisted, please check your blacklist settings on your account on the site
  • an old test e-hentai login script from 2018 that is no longer in the client defaults will be deleted from clients that still have it today. if the user has no other login script for e-hentai, the domain entry will be deleted as well. this removes potential technical barriers for users that wish to use hydrus companion to access e-hentai, which is now the recommended method
  • hydrus mpv now has an appropriate stream title, which propagates up to the os-level sound mixer. it was previously the ugly hydrus filename
  • improved error handling when mpv is passed an invalid conf
  • the default mpv conf now has audio normalisation that seems to work ok
  • fixed an issue with the 'delete/move out missing/corrupt file' file maintenance job where record deletes were not processing correctly. it now deletes the file record correctly and also clears that deletion record, to make re-import of the correct file, if found, easier
  • all hydrus menu labels are now "middle...elided" when they are greater than 64 characters
  • all new hdd, url, and simple download pages should now obey the 'remove files when trashed' rule. pages in existing sessions will not
  • updated the user-created CutieDuck darkmode qss file to the latest version, which alters the recent hydrus qss styling colours like green/red button labels
  • did a full pass of all service fetching--all file and tag services should now present in lists and tabs in service_type, alphabetical order, e.g. for manage tag siblings, the tabs will always be local_tags, tag_repositories, both in alphabetical order
  • fixed an issue where a 'get darker or lighter comparison colour' calculation was not working well for black or very dark colours
  • if subscriptions or general network traffic is paused, the bandwidth section of the main gui statusbar now says it
  • the status bar now tooltips each section
  • clarified some labels on the edit url class panel
  • moved all delayed focus-shifting code to a more stable system
  • cleaned up how the global icon cache is initialised and referenced
  • updated the hydrus project gitignore to hide all db, log, server, recovery, and media files that could be under the db directory
  • updated the endchan links in the help to have a .org secondary link
  • more general code refactoring

• version 387

  • the sort-files-by dropdown is now a button that launches a nested menu. it still supports mouse wheel events. it should now be quicker to find what you want!
  • added 'sort by framerate' to regular file sort. it works for file search at the db level as well, when mixed with system:limit
  • under options->sort/collect, the namespace sort-by ui has finally had its makeover. it now has add/edit/delete buttons and up/down buttons for reordering how the entries will appear. it also deals with bad input better. furthermore, namespaces that have hyphens (like 'creator-id') are now supported in namespace sort (and hence collect-by dropdowns!)!
  • numerical (multi-star) ratings can now be set by dragging the mouse across the line of stars
  • added 'duplicate page' to the page tab right-click menu! it just makes a copy of the page or page of pages right beside it
  • system:everything will now always show up in non-query-page autocomplete dropdowns (such as in the file maintenance dialog)
  • wrote a maintenance routine to repopulate and correct the tag text search cache. it is possible to trigger this (though it is typically pointless) from the database->maintain menu
  • updated the characters that are ignored in autocomplete tag text search rules, which help skip over unusual characters and assist word-break discovery for searching for tags like '[intensifies]'. as well as the previous brackets, braces, paretheses, quotes, and double-quotes, now slash, backslash, hyphens, and underscores(!) are ignored. searching for 'bbb' will now match a tag 'aaa-bbb', and searching for 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'!
  • to effect the above change, the client will take a few seconds to a minute to update
  • the above tag text search rules now collapse contiguous unusual characters, or combinations of whitespace and characters, better
  • namespace and simple wildcard search inputs no longer have the tag text search rules applied to them, meaning you can now search for these unusual characters more specifically when desired
  • updated the derpibooru gallery search objects to use their api, thanks to a user's submission. this re-enables the 'no filter' mode
  • added watcher support for tvch.moe, which works with an existing 4chan-style parser
  • the 'add the ptr' help item now warns the user about the ptr's modern drive storage requirements (4GB download+files, 25GB db). the help files are also updated
  • I believe I fixed the sometimes crazy fast media drag-move that could happen in archive/delete and duplicate filters
  • fixed an old uncaught wx->qt issue with the simple downloader where editing the formulae would throw an error
  • fixed a bug in the 'move highlighted thumbnail' code in the rare case where the currently focused thumbnail can not be found
  • text input dialogs are now mostly wider
  • refactored some ui code, cleaning up core objects and import hierarchy
  • did some controller/gui refactoring, pushing on untangling things
  • cleaned up a bunch of no-longer-used import statements
  • misc ui code cleanup
  • slight rewording of database menu
  • prepped shortcuts system to ignore a window-activating click (for the media viewer filters), but can't turn it on yet as media viewer clicks are not yet fully plugged in

• version 386

  • gifs and mpv

  • the client now parses gifs for loop count metadata (some gifs say they should only be played x times through, usually 1). options->media now has a checkbox to control whether this value should be obeyed. both the native viewer and the mpv viewer should follow this. default value is still to loop indefinitely
  • if gifs are set to play with the mpv player, those without duration will now still be loaded in the native image viewer. the media viewing options ui now notes this
  • the mpv.conf file used in the mpv window can now be changed under options->media. it _should_ update the conf for all open mpv players on options dialog ok. added to the hydrus static mpv-conf directory are three new 'test' mpv confs for high quality and two audio normalisation tests. all test feedback and recommended conf info is welcome

  • ui cleanup and improvements

  • the media viewer mouse autohide time is now customisable under options->media, including disabling it completely. it defaults to 700ms
  • improved the timing and reliability of the media viewer mouse autohide code
  • the mouse should now never autohide while a dialog is open
  • improved the bad colours of the splash screen. it should now be all one colour now, no ugly stand-out white square or other hardcoded colours. hydev also deployed his unparalleled gimp skills to get a white fade around the transparent-background hydrus icon, so it should look correct in darkmodes as well
  • created a default_hydrus.qss file in the qss folder in order to handle formerly hardcoded colours using hydrus-specific classnames and properties. as well as being loaded by default, this qss file is prepended to any custom stylesheet, so any custom stylesheet that includes its own versions of the hydrus-specific entries will override the defaults. this qss will get more work in future
  • added on/off buttons to hydrus default qss and converted existing object to use class and properties to obey this
  • added a variety of valid/invalid/warning text colours to hydrus default qss and converted existing text objects to use classnames to obey this
  • added accept/cancel buttons to hydrus default qss and converted all green/red buttons across the program to use classnames to obey this
  • the migrate database dialog now has an outright 'remove location' button to reduce confusion and speed up removal of high weight locations
  • if a location does not exist on the migrate database dialog, it will now stop throwing multiple error popups every time the list slightly changes, and will complain if file rebalancing is attempted, and will provide different 'remove' yes/no messages if that missing location currently has files or not
  • slight ui touch-ups to the migrate database dialog
  • if a window that remembers its position attempts to re-position to a location not on a current display, the windowing system now attempts to rescue it to the primary display, with appropriate popup messages given and errors caught more gracefully
  • extended these off-screen rescue calls to windows that pull their position from their parent. e.g. if you open the options dialog while the main gui is half over the left side of your screen, it should rescue to the primary display
  • windows that position off the center of their parent now calculate that reliably on the parent window, not just the parent widget, which never really worked as intended
  • windows that have no position memory and no parent to pull center/topleft position from will now appear center/topleft of the monitor your mouse is on
  • the splash screen now appears centered on the monitor your mouse is on
  • cleaned up and improved a bunch of window/screen coodinate code, moving 'space on screen' calculations to 'space on screen minus taskbar' and similar
  • unified a 'dialog is open' check across the program
  • cleaned up the old wx->Qt size, coordinate, and colour conversion code
  • cleaned up some old wx->Qt calculation code
  • improved 'light' and 'grey' colour detection code to now work in HSV
  • improved colour changing code to now work in HSV
  • improved some internal single-shot scheduled job code

• version 385

  • mpv

  • the mpv window is now plugged into the slideshow system, so when an mpv window has played its media once, a media viewer browser currently slideshow-ing will now correctly know to move to the next media
  • slideshows of videos should progress to the next file faster and more smoothly, with no more half-second blit of the start of the movie before the next file loads
  • pausing video/audio is no longer cause for the slideshow to move on--now only the 'has played once through' check will naturally trigger it. you can now more reliably seek/scrub a video during slideshow
  • the mpv window and native animation player is now better about pausing while a seek drag (scrub) is ongoing
  • various music videos that have 1 frame should now show a seekbar correctly for the mpv window, with correct timecode based seeking
  • fixed a bug in the video/audio resizing code that meant zoom in/out cycles would move a video player down a few pixels off center
  • the blank audio mpv window now has basic hardcoded zoom support and will scale down to a too-small viewer window, so you can still access the seek bar
  • fixed some 'has duration' calculations for audio/video that has duration but no frame count

  • shortcuts

  • if a shortcut handler on an individual window does not have a double-click mapping for a command--and furthermore if none of its parent windows that are fully plugged in have one either--it will attempt to map the single-click version of the event as a backup. so now if you have an archive/delete filter, you can click fast again and the double-clicks will be interpreted as single-clicks (unless you map double-click to mean something else on the media_window or one of the media_viewer parents)
  • the media viewers across the program are now fully plugged into the shortcuts system for key presses, and half-plugged in for mouse clicks
  • 'close_media_viewer' is added to all 'media_viewer' shortcut set types. enter/return/escape are now defaults for 'media_viewer' (applying to all), and middle-click/double left-click are now defaults for 'media_viewer_browser'. this is no longer hardcoded. if you are a madlad, you can now unmap all 'close_media_viewer' commands
  • double left-click is now assigned to 'keep' in the 'archive_delete_filter' shortcut set. due to the new click/double click rules above, this means that by default double clicking a video/audio in the archive/delete filter will now mean 'keep and move on' on that fast second click!
  • edit shortcut set ui now sorts its command list on load
  • significant shortcuts refactoring
  • general shortcuts code and debug code cleanup and improvement

  • the rest

  • local file import pages and most downloaders will now report more file import steps to their status labels. most will blit by too fast to see, but if it hangs for a bit, you will now see the step it is caught up on. I imagine in most cases, this will be metadata generation for large videos
  • fixed a variety of searches that could return files not filtered to the current file domain (e.g. files in trash while in my files, or not in ipfs while in ipfs) when the search did not include an inclusive tag
  • updated the default danbooru file page parsers to get their new tag format, thank you to a user for submitting these
  • a popup message now appears while sessions are loading. it auto-dismisses once the load is complete
  • the edit media view options dialog (under options->media, launched from the filetype list) is now better at disabling non-applicable widgets based on filetype
  • fixed an issue where clicking from the autocomplete dropdown floating window to the same control's text input could result in a single flicker-frame where the dropdown is hidden
  • tightened the size of the splash screen white background. figuring out appropriate colour from the current stylesheet remains elusive lmao
  • cleaned up and wx->Qt converted a variety of event handling code
  • updated some 'mime' labels across the program to the new 'filetype' wording

• version 384

  • shortcuts

  • the shortcut system now supports mouse double-clicks--left, right, or middle
  • the shortcuts system now differentiates between press or release single mouse clicks--although complete support for release mouse events may be a bit patchy, as full mouse integration is ongoing
  • the shortcut edit ui is now simpler--the command type is selected by a list, and the individual command sub-panels hide and show as appropriate--no more stupid 'set command' buttons
  • the shortcut edit ui now has a 'restore defaults' button that will restore an individual set back to default settings
  • two new shortcut sets are added--'media_viewer_media_window' and 'preview_media_window'. they control pause, pause/play, open_externally, and close/launch_media_viewer respectively. they work on the static image viewer, the native animation widget, and the new mpv player, and they support mouse clicks. the old pause/play (formerly left-click) and open_externally (double left-click) commands are no longer hardcoded
  • by default, the preview window's media window now launches the media viewer on a middle- or double-left-click
  • 'media_viewer_browser' shortcut set now has 'release right-click' bound to 'show_menu', a new command, which is no longer hardcoded
  • most menus across the program can now be opened with the keyboard context menu key
  • the 'global' shortcut set now has 'exit_application', 'exit_application_force_maintenance', and 'restart_application' commands
  • fixed the rating increment/decrement command option not hiding in non-'media' shortcut sets
  • fixed some issues loading edit ui for shortcuts with rating actions
  • significant refactoring and some cleaning of shortcut code

  • the rest

  • mpv windows should not longer get a single frame of previous-window-stretch when flicking between one mpv media to another with a different aspect ratio on the same media canvas. when a video is caught in a frame of loading, it should now flicker a frame of black
  • switching from one static image or native animation to another of the same type _should_ be less likely to do a single frame of stretch when transitioning. when an image or animation transition is caught on a new frame, it _should_ now flicker a frame the same colour as the media canvas background
  • the string transformation edit panel's individual transformation rule edit panel has had some more work: much like with shortcuts, the controls now hide and show based on transformation type, the controls' text labels now change based on transformation type, and the example text now updates on any widget change. the manual 'update example' button is removed
  • fixed a typo that caused an error when establishing the correct mouse cursor to use over the volume control when hydrus was using PyQt5 (rather than PySide2)
  • in order to reduce accidental micro-drags that cause mpv load-pause issues, starting a thumbnail drag now takes more pixels and requires a smoother drag to start, let's see how it goes
  • improved the show/hide logic of the floating autocomplete dropdown window. it should now also reliably detect when window focus goes from the dropdown itself to another window
  • fixed a bug where clearing the deletion record of a deleted tag would not remove the record from the fast cache that populates thumbnail tags (making it seem on most file loads that the tag still existed). if you were hit by this previously, please hit _database->regen->a/c cache_ one time to resync the cache
  • relatedly, thumbnails should now correctly live-update their deleted tags on deletion record clearance updates
  • if mpv is not available, opening the about window will now make a popup with the actual import error trace
  • significant refactoring of various ui code

• version 383

  • mpv

  • updated the prototype volume/mute controls on the top media viewer hover window to be a proper 'speaker' icon button for mute with a volume slider that pops up or down on mouse-over
  • the new volume control is on the hover window and any media that has audio
  • the right-click menu of the preview viewer and media viewer now have volume submenus to set mute/volume

  • the client now has multiple volumes and mutes

  • for mute, there is a global mute which overrides everything, and the preview and media viewers have their own mutes that just apply there.
  • under options->audio, you can choose whether preview windows have their own separate volume value, default is yes they do
  • there is a new shortcut set called 'global', which applies on the main gui and the media viewer both, and which currently has actions to alter global mute. by default, ctrl+g flips global mute
  • after reports of unusual rendering bugs for some users, the default mpv.conf is now more barebones. more work will happen here

  • linux

  • the linux release is now built on Ubuntu 18.04 (was 16.04). unfortunately, my build packager bundled in a variety of surplus libraries, so the archive has bloated somewhat--I have removed some that I am confident are not needed, but I may have made a mistake, and there are likely more that can be taken away
  • the linux release now comes with mpv support
  • please let me know if you have any errors running this build or loading mpv. early tests seem good though!

  • the rest

  • the launch/exit splash screen now uses a cleaner Qt-compatible layout system. It resizes and obeys stylesheets better, colouring text and background according to current style
  • removed the 'has duration' text label option from 'audio and duration' options panel as it is no longer used, and renamed the panel back to just 'audio'
  • the string transformation edit panel's individual transformation edit panel now shows that transformation step's example string and the transformed string, which is updated by button. this edit panel will get some more love soon, including dynamic hide/show of applicable controls and live updates of the example transformation as you type
  • misc ui layout improvements
  • misc ui improvements

• version 382

  • mpv

  • rolled Qt back from 5.14.0 to 5.13.0 on the releases, which seems to have fixed our 'event queue sometimes halts until mouse move' issue that occurs after initial mpv load. some other ui and media viewer resize jank seems to be cleared up. I dunno what happened with 5.14, and I don't suspect it as the problem nearly as much as my currently borked Qt event processing code, but rolling back seems the easiest solution for now
  • fixed an issue that was crashing non-windows that were able to load mpv
  • mpv now loads an mpv.conf from install_dir/static/mpv-conf. please feel free to swap in another conf or edit that one as you like. I would be interested in feedback
  • default mpv conf is now set to specifically enable some hardware acceleration to improve playback for some users, and to never load sidecar files like subtitles as this was introducing incredibly large load lag for users with large/high latency file storage
  • fixed a new issue where preview windows were not unloading media (particularly significant for mpv with audio) on page change and client shutdown
  • fixed an issue with global volume propagation to multiple open mpv widgets

  • the rest

  • added two dark qss stylesheets from the user-creation github repo to the default install
  • when zooming out from a zoom that makes the media bigger than the media viewer canvas to a zoom where it fits, the media will now recenter. see if you like this, maybe it should be an option
  • to help forestall unnamespaced filename tag spam in various new-user scenarios, the 'filename' checkbox-and-namespace widget on the filename tagging options panel now initialises with 'filename' as the namespace
  • fixed a recent window sizing issue with the 'the client is already running' dialog not appearing
  • file export paths that include subdirectories that could possibly have empty text, like "[creator]/[page]", will no longer error when this is so (e.g. if a file in this case has no creator tags)--they will eliminate the subdirectory entirely, becoming "[page]". this should work for all platforms and for any nested subdirectory
  • fixed an issue with some fractional dataspeeds below 1KB/s displaying with many significant figures
  • improved some custom event handling definition code
  • reworked hydrus's internal object publisher/subscriber messaging system to be more Qt-happy
  • if the file import tagger is given a neighbouring .txt file to pull tags from that does not decode to utf-8 nicely, it now catches and reports the error more gracefully
  • reworded a bit of the installing help and first-start popup to emphasise that hydrus does not auto-update
  • added links to https://github.com/Zweibach/text/blob/master/Hydrus/PTR.md , a new guide for the PTR, to the help
  • removed the old 'hardcoded shortcuts' help entry, since it is increasingly irrelevent

• version 381

  • mpv

  • mpv is now available and the default for all windows users
  • I believed I have eliminated the final reported mpv crash
  • mpv load and unload delays are greatly reduced. initial load still takes about half a second, but subsequent loads are now as quick as native renderers
  • mpv seems to work well for gif and apng
  • added a very simple global volume slider and audio mute checkbox to the media viewer top hover window. this was a quick patch--much better controls and shortcuts will come in future
  • mpv windows now properly re-show the cursor on mouse movement
  • unified mpv mouse press/release handling with native animation--click down now does pause/play and starts a drag event
  • unfortunately, in some cases embedding mpv requires overriding local OS number rendering (e.g. 1,234 vs 1.234). hydrus number rendering is now coerced to the english style with commas until we can figure out a better solution--sorry!
  • cleared up an issue where simple clicks on page tabs would trigger micro-page drags that were immediately cancelled. this situation was exacerbated when the page being left had an active mpv window. the flicker of page drag cursor is now gone, and some weird situations where static clicks during busy time could move a tab should be fixed
  • eliminated the recent issue in the media viewer where transitioning from one media type to another through navigation, particularly mpv->other, would flicker a single frame of the last 'other' media shown(!)
  • fixed a bug where repeated mpv views in the preview viewer could disable client file drag and drop
  • the bug where thumbnails may not waterfall in unless the mouse is moving after some mpv videos are loaded for a page is relieved but not completely fixed
  • if the preview window is collapsed and hidden, media will no longer ever load into it
  • fixed an edge-case bug where the mpv window would not like being told to show nothing when it was already showing nothing
  • wrapped mpv load errors in a basic graceful catch
  • fixed an issue some users had with loading mpv's dll

  • file types

  • a new file metatype, 'animation', is added, for gif and apng. these are no longer considered 'image' for a variety of purposes
  • the filetype selection panel, which is used in system:filetype and import folder UI, has had an overhaul--it now has tristate 'mime group' checkboxes to represent a half-filled group and expand/collapse buttons to hide the tall filetype lists. individual filetype lists will start hidden unless their default value is a partially filled group
  • the media view options have a similar overhaul: they are now collapsed to general filetypes by default. you set view and zoom options for the generalised 'video' type under options->media, and if you want to set specific options for webm or anything else, you can add/delete those types to override the general default
  • the new default options for a fresh client are just for these general types. if mpv is available, video, animations, and audio now start with mpv as the default viewer. video and animation zoom is now flexible (not fixed to 50%, 100%, 200%) and will fill the media canvas
  • all media view options will be reset to this simple default on update! if you have specific zoom or display preferences, please reset them after the update--but you might like to play with mpv a bit first, as it renders at large and smooth zooms very well

  • the rest

  • the new thumbnail right-click file selection routine will now only focus and scroll to the first member of the selection if no other members of the new selection are already in view
  • fixed some caching code and sped up the new select/remove menu count generation (which can lag for very large pages) by two to six times
  • sped up file filter counting code by about ten percent
  • fixed weird layout on: migrate database panel, duplicates page (left and right), edit shortcuts, edit import folder, and the filename tagging panel
  • fixed an issue where the media viewer's hover windows might flicker into view for one frame when the mouse moved over the center of the media viewer for the first time
  • fixed a media viewer shutdown issue that would sometimes lead to the first file in the list being opened in the shutting-down viewer for an instant or highlighted as the new thumb focus
  • the file maintenance system that queues up missing/broken files' urls for redownload will no longer re-select the download page on every new url
  • fixed an issue where a downloader's tag blacklist was not being applied on the child files of certain kinds of multiple-file post (such as with pixiv)
  • deleting a very long tag should no longer create a very wide confirmation dialog in the manage tags dialog
  • fixed some 'the panel grew a bit, but the parent window didn't grow quite enough and now it has scrollbars for two pixels of extra content' sizing issues
  • fixed some dialog sizing calculations when the parent window was borderless fullscreen
  • maybe fixed a rare event processing bug
  • improved quality of some misc data comparison code across the program
  • did some significant backend event/pubsub code cleanup, mostly related to getting mpv working a bit cleaner
  • improved thumbnail rendering time
  • improved smoothness of thumbnail fade animations (at least for when they are working right, ha ha!)
  • misc fixes

• version 380

  • basic mpv support is added. it comes with the windows build this week, and is a prototype meant for initial testing. the library is optional. users who run from source will want 'python-mpv' added via pip and libmpv available on their PATH, more details in running_from_source help
  • took an qt-mpv example kindly provided by a user, updated it to work with the hydrus environment, and integrated it into the client as a new choosable view type under audio/video filetypes under options->media for advanced users
  • reworked how the 'start paused' and 'start with embed button' media viewer options work under options->media. these are now separate checkboxes, not combined with the underlying 'show action'. existing embed/paused show actions should be converted automatically to the correct new values
  • unfortunately, due to some python/qt/libmpv wrapper mouse interaction issues, mpv's 'on screen controller' overlay is not available
  • for now, left click pause/plays the mpv window, just like the native mpv window.
  • preview/next frame shortcuts should work for the mpv window when playing video
  • no volume/mute controls yet, these will come in the coming weeks, including global mute settings
  • updated media show and sizing code to account for mpv widgets
  • reworked my animation scanbar to talk to mpv, and for my mpv window to talk back to it
  • improved the animation scanbar to be more flexible when frame position and num_frames are not available, both in displaying info and calculating scanbar seek clicks
  • mpv api version added to help->about

  • new downloader objects

  • thanks to a user, updated the 'pixiv artist page' url class to a new object that covers more situations. the defunct 'pixiv artist gallery page' url class is removed
  • added 8kun and vch.moe download support. I got started on julay, smug, and endchan, but they were a little more tricky and I couldn't finish them in time--fingers crossed, next week

  • menu quality of life

  • a right-click on thumbnail whitespace will now not send a 'deselect all' event! feel free to right-click in empty space to do an easy remove->selected

  • remorked the tag menu layout to move less frequently used actions down

  • - moved the discard/require/permit/exclude search predicate actions down
  • - moved 'open in a new page' below select and copy
  • - moved copy above select
  • and some misc menu layout improvement on this menu
  • fixed some labelling with the discard/require/permit/exclude verbs on negated tags
  • right-clicking on system search predicates now shows the 'copy' menu correctly
  • system predicates that offer easy inverse versions (like inbox/archive) should now offer the 'exclude' verb
  • when right-clicking on a single tag that has siblings, its siblings and those siblings' subtags will now be listed in the copy menu!
  • copying 'all' tags from a list menu, with or without counts, will now always copy them in the list order
  • across the program, all menu 'labels' (menu text items that do not have a submenu and have no associated action, like 'imported 3 years 7 months ago') will now copy their text to the clipboard. let's see how it goes

  • other ui quality of life

  • across the program's UI, filetypes are now referred to with simpler terms rather than technical mimetypes. instead of 'image/jpg', it is now typically just 'jpeg'
  • the 'remove selected' buttons on the gallery and watcher pages are now smaller trash icon buttons
  • the new page chooser will now auto-dismiss if it loses focus--so if you accidentally launch it with a middle-/double-click somewhere, just click again and it'll go away
  • hitting enter or return on the new page chooser now picks the 'first' button, scanning from the top-left. hitting enter twice now typically opens a new 'my files' search page
  • added pause_media and pause_play_media shortcuts to the media_viewer shortcut set. new clients will start with space keypress performing pause_play_media
  • added pause_play_slideshow shortcut to the media_viewer_browser shortcut set. this shortcut is no longer hardcoded by space keypress
  • the six default shortcut sets now have a small description text on their edit panels
  • the options->media edit panels now enable/disable widgets better based on current media/preview action
  • added a checkbox to _options->gui pages_ to set whether middle-clicking a tag in the media viewer or a child tag manager to open a tag search page will switch to the main gui. default is false
  • mr bones now reports total files, total filesize, and average filesize
  • mr bones now loads your fate asynchronously

  • the rest

  • added tentative and simple realvideo (.rm) and realaudio (.ra) support--seems to work ok, but some weirder variable bit rate formats may not, and I have collapsed the various different extensions just down to .rm or .ra
  • added trueaudio (.tta) audio support
  • fixed a bug from the recent search optimisations where a bare inbox search would not cross-reference with the file domain (so some trash could show up in a simple inbox/'my files' query)
  • fixed an issue with searching for known urls by url class where the class was for a third-or-higher-level domain and was not set to match subdomains (this hit 4chan file urls for a few users)
  • fixed the issue with 'open externally' button panel not clearing their backgrounds properly
  • fixed some of the new unusual stretchy layouts in the options dialog
  • removed overhead from subscriptions' 'separate' operation, which should stop super CPU hang when trying to split a subscription with hundreds of thousands of urls
  • fixed an issue where the advanced file delete dialog would not show the simple 'permanent delete' option when launched from the media viewer's right-click menu
  • fixed the select/remove actions for local/remote
  • fixed 'set_media_focus' from manage tags to correctly activate the underlying media viewer as well as set focus
  • stopped the 'file lookup script' status control from resizing so wide when it fetches a url
  • fixed a rare mouse wheel event handling bug in the media viewer
  • reduced db overhead of the 'loading x/y' results generation routine. this _may_ help some users who had very slow media result loading
  • cleaned up how the server reports a bootup-action error such as 'cannot shut down server since it is not running'--this is now a simple statement to console, not a full error with trace
  • improved client shutdown when a system session shutdown call arrives at the same time as a user shutdown request--the core shutdown routine should now only occur once
  • fixed an issue with thumbnail presentation on collections that have their contents deleted during the thumbnail generation call
  • misc wx->Qt layout conversion improvements
  • updated the github readme to reflect some new links and so on
  • misc code cleanup

• version 379

  • downloaders

  • the right-click menus from gallery and watcher page lists now provide a 'remove' option
  • gallery and watchers now provide buttons and menu actions for 'retry ignored'
  • activating a file import status list (double-clicking or hitting enter on a selection of rows) now opens the selection in a new page
  • file import status buttons now have show new/all files on their right-click menus
  • on gallery and watcher pages, the highlight, clear highlight, pause files, and pause search/check buttons are now smaller bitmap buttons
  • as the old default pixiv login script is completely broken, any client with this active will have it deactivated and receive an update popup explaining the situation and suggesting to use Hydrus Companion for login instead
  • updated the derpibooru downloader

  • search

  • when search predicates are added to the active search list, they are now better able to remove existing mutually exclusive/redundant predicates

  • - system:limit, hash, and similar to predicates now remove other instances of their type
  • - system:has audio now removes system:no audio and vice versa
  • - any search predicate will remove system:everything (see how you feel about this)
  • improved 378's db optimisation to do tag searches in large file domains faster
  • namespace search predicates ('character:anything' etc...) now take advantage of the same set of temporary file domain optimisations that tag predicates do, so mixing them with other search predicates will radically improve their speed
  • wildcard search predicates, which have been notoriously slow in some cases, now take full advantage of the new tag search optimisations and are radically faster when mixed with other search predicates
  • simple tag, namespace, or wildcard searches that are mixed with a very large system:inbox predicate are now much faster
  • a variety of searches that include simple system predicates are now faster
  • integer tag searches also now use the new tag search optimisation tech, and are radically faster when mixed with other search predicates
  • system:known url queries now use the same temporary file domain search optimisation, and a web-domain search optimisation. this particularly improves domain and url class searches
  • fixed an issue with the new system:limit sorting where sort types with non-comprehensive data (like media views/viewtime, where files may not yet have records) were not delivering the 'missing' file results
  • improved the limit/sort_by logic to only do sort when absolutely needed
  • fixed the system:limit panel label to talk about the new sorted clipping
  • refactored tag searching code
  • refactored namespace searching code
  • refactored wildcard searching code and its related subfunctions
  • cleaned all mappings searching code further

  • the rest

  • m4a files (and m4b) are now supported and recognised as separate audio-only mp4 files. files with a single jpeg frame for their video stream (such as an album cover) should also be recognised as audio only m4a for hydrus purposes for now. better single-frame audio support, including functional thumbnails and display, is planned for the future. please send in any m4a or m4b files that detect incorrectly
  • the remove thumbnail menu has been moved to a new, cleaner file filtering system. it now presents remove options for different file services and local/remote when available (most of the time, this will be 'my files'/'trash' appearing when there is a mix), including with counts for all options
  • the select thumbnail menu is also moved to this same file filtering system. it has a neater menu, with counts for each entry. also, when there is no current focus, or it is to be deselected, the first file to be selected is now focused and scrolled to
  • for thumbnail icon display and internal calculations, collections now _merge_ the locations of their members, rather than intersecting. if a collection includes any trash, or any ipfs members, it will have the appropriate icon. this also fixes some selection-by-file-service logic for collections
  • import folders, export folders, and subscriptions now explicitly only start after the first session has been loaded (so as well as freeing up some boot CPU competition, a quick import folder will now not miss publishing a file or two to a long-loading session)
  • the subscription manager now only waits 15s before starting first work (previously, the buffer was 60 seconds)
  • rearranged migrate tags panel so action comes before destination and added another help text line to clarify how it works. the 'go' confirmation dialog now summarises tag filtering as well
  • tag filter buttons now have a prefix on their labels and tooltips to better explain what they are doing
  • the duplicate filter right-center hover window should now shorten its height appropriately when the pairs change
  • fixed a couple of bugs that could appear when shutting down the duplicate filter
  • hackily 'fixed' an issue with duplicates processing that could cause too many 'commit and continue?' dialogs to open. a better fix here will come with a pending rewrite
  • dejanked a little of how migrate tags frame is launched from the manage tags dialog
  • updated the backup help a little and added a note about backing up to the first-start popup
  • improved shutdown time for a variety of situations and added a couple more text notifications to shutdown splash
  • cleaned up some exit code
  • removed the old 'service info fatten' maintenance job, which is not really needed any more
  • misc code cleanup
  • updated to Qt 5.14 on Windows and Linux builds, OpenCV 4.1.2 on all builds

• version 378

  • if a search has system:limit, the current sort is now sent down to the database. if the sort is simple, results are now sorted before system:limit is applied, meaning you will now get the largest/longest/whateverest sample of the search! supported sorts are: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. this does not apply to searches in the 'all known files' file domain.
  • after identifying a sometimes-unoptimal db access routine, wrote a new more reliable one and replaced the 60-odd places it is used in both client and server. a variety of functions will now have less 'spiky' job time, including certain combinations of regular tag and system search predicates. some jobs will have slightly higher average job time, some will be much faster in all common situations
  • added additional database analysis to some complicated duplicate file system jobs that adds some overhead but should reduce extreme spikes in job time for very large databases
  • converted some legacy db code to new access methods
  • fixed a bug in the new menu generation code that was not showing sessions in the 'pages' menu if there were no backups for these sessions (i.e. they have only been saved once, or are old enough to have been last saved before the backup system was added)
  • fixed the 'click window close button should back out, not choose the red no button' bug in the yes/no confirmation dialogs for analyze, vacuum, clear orphan, and gallery log button url import
  • fixed some checkbox select and data retrieval logic in the checkbox tree control and completely cleared out the buggy ipfs directory download workflow. I apologise for the delay
  • fixed some inelegant multihash->urls resolution in the ipfs service code that would often mean a large folder would lock the client while parsing was proceeding
  • when the multihash->urls resolution is going on, the popup now exposes the underlying network control. cancelling the whole job mid-parse/download is now also quicker and prettier
  • when a 'downloader multiple urls' popup is working, it will publish its ongoing presented files to a files button as it works, rather than just once the job is finished
  • improved some unusual taglist height calculations that were turning up
  • improved how taglists set their minimum height--the 'selection tags' list should now always have at least 15 rows, even when bunched up in a tall gallery panel
  • if the system clock is rewound, new objects that are saved in the backup system (atm, gui sessions) will now detect that existing backups are from the future and increase their save time to ensure they count as the newest object
  • short version: 'remove files from view when trashed' now works on downloader thumbs that are loaded in from a session. long version: downloader thumb pages now force 'my files' file domain for now (previously it was 'all local files')
  • the downloader/thread watcher right-click menus for 'show all downloaders xxx files' now has a new 'all files and trash' entry. this will show absolutely everything still in your db, for quick access to accidental deletes
  • the 'select a downloader' list dialog _should_ size itself better, with no double scrollbars, when there are many many downloaders and/or very long-named downloaders. if this layout works, I'll replicated it in other areas
  • if an unrenderable key enters a shortcut, the shortcut will now display an 'unknown key: blah' statement instead of throwing an error. this affected both the manage shortcuts dialog and the media viewer(!)
  • SIGTERM is now caught in non-windows systems and will initiate a fast forced shutdown
  • unified and played with some border styles around the program
  • added a user-written guide to updating to the 'getting started - installing' help page
  • misc small code cleanup

• version 377

  • qt

  • all non-menubar menus across the program now launch on click release. some previously launched on click press. a variety of related click event behaviour is cleaned up, particularly with thumbnail/tag selection on the click down. this also fixes some users' menus immediately activating the first entry on slow clicks in some ui styles
  • I think I fixed the annoying single-frame delayed size-down resize on media viewer hover frames when changing media!
  • the vast majority of old wx panel background colour hacks are removed, so custom stylesheets should now cover much more of the UI
  • improved the new custom style and stylesheet setting, resetting, and error handling code, particularly for not re-applying the same style or stylesheet twice, and for handling un-re-settable styles (seems to be defaults initialised by third-party OS-wide Qt style) gracefully
  • fixed hyperlinks not using the custom web browser launch path as set in the options
  • fixed the 'migrate entire db' and 'set thumb location' buttons in the migrate database dialog
  • fixed a typo bug when launching the url selection tree after adding an ipfs directory to download
  • fixed two typo bugs when editing regex favourites and simple downloader formulae
  • fixed an issue where custom shortcut sets could not be deleted
  • fixed a typo in the edit account type panel
  • fixed sorting the login listctrl when there are session logins mixed with non-session logins
  • removed some old media viewer hover window display/raise hacks
  • retired the 'always show hover windows' debug mode
  • the media viewer will no longer perform any drag calculations on anything but left-click drag
  • misc Qt code refactoring/cleanup

  • url searching

  • the database now stores 'known url' domain information more efficiently. it will take a few moments/minutes to reshape the db when updating
  • system:known url's exact url search now runs extremely fast. this will only affect new predicates of this type, not those in existing sessions
  • system:known url's domain search now runs much faster and matches subdomains of the given domain. this will only affect new predicates of this type, not those in existing sessions
  • system:known url's url class search now runs much faster. this will only affect new predicates of this type, not those in existing sessions
  • when entering a regex system:known url predicate, the dialog will now not OK (throwing up an error dialog) if the regex is invalid

  • the rest

  • the shortcut system now allows all text characters. if it has text, it should work, but it is the wild west in terms of modifier labelling. anything unusual on your keyboard like ctrl+alt+e to make æ will _display_ as ctrl+alt+æ, but the same key combination will match up in the program all correct
  • added shortcut actions 'pan_top_edge', 'pan_bottom_edge', 'pan_left_edge', 'pan_right_edge' to the media viewer shortcut set that will move the current image so the respective edge is aligned with the larger canvas's
  • added shortcut actions 'pan_horizontal_center' and 'pan_vertical_center' to do as above but center on that axis
  • session save now hangs the UI significantly less, whether triggered by user command or auto-saving 'last session'
  • saving of last/exit sessions on client close is a little faster
  • the call to refresh thumbnail file info (and redraw if needed) when a file is imported or has metadata-regenererating file maintenance done will now only call for files that are actually loaded, run faster per file, run faster when the client has large collections in its session, and not hang the ui thread when waiting for the new media info to arrive
  • like regular popups, modal popups (like those created when big vacuum/analyze jobs jump in) will now only appear if the main gui or an on-parent child has OS focus
  • the main gui/on-parent child OS focus test now includes misc child windows like the autocomplete results hover window
  • network jobs that fail for one reason or another will now be more reliably cleaned up, and their connections returned to the connection pool. this may fix the 'too many open file handles' errors some users were seeing after long term unreliable network traffic
  • fixed an issue where some thumbnails that were trashed or physically deleted were being removed from 'all known files' and file repository views when it was not appropriate
  • connection and downloader retry time options now have a wider min/max range when in advanced mode, with an accompanying warning label for the connection panel
  • checker options times now have a wider min/max range when in advanced mode, with an accompanying warning label
  • cleaned up some shutdown reporting text
  • misc debug improvements

• version 376

  • subscriptions

  • wrote a new subscription manager to better look after subscription scheduling
  • rather than checking every four hours or after manage subs dialog close, subscriptions now record an indication of when they are next due for work, whether that is the estimated next check time or when bandwidth is free on remaining file downloads, and launch in a fifteen-minute window around that time. delays due to previous errors or user cancels are also taken into account. this reduces background cpu and i/o greatly for clients with large subs
  • if a sub is paused, or all its queries are paused, it will now never be reloaded after first load until a change via the manage subs dialog
  • furthermore, if a single sub takes a very long time to work, the whole sublist can re-cycle if they come up due for more work before it is finished
  • if a sub query is DEAD but still has outstanding files to download, it will no longer automatically pause
  • subs now clean up more tidily if they are running on a program exit
  • the subscription popup now shows check/file progress based on the number of queries that appear to have pending work. instead of 'query 300/450' with 420 that aren't due, you'll get 'query 12/30'. if a query becomes due during a round of checking, another round of checking will run
  • if a subscription fails to load from the db, the error is handled better and no more subs will run in that boot
  • improved subscription startup checking logic, tightening up various paused/dead/cansync tests
  • improved subscription interrupt checking logic, tightening checks on global network pause and various shutdown scenarios
  • cleaned up some more subscription code in prep for data storage breakup

  • qt

  • added experimental Qt style settings to the new options->style page! all users should now be able to set Fusion style, and perhaps some alternate OS styles. advanced users are invited to play around with QSS stylesheets (although be warned that some of hydrus's custom colour system overrides QSS, so more work is needed here), which will be extended and made user-friendly in coming weeks
  • fixed tab position calculations for all tab/media drag and drops for tab bars that are centered or otherwise positioned far off top-left alignment
  • fixed tab drag and drop event object handling for macOS. tab and media DnD is now enabled for macOS
  • the popup toaster can now unhide if an on-top-of-parent non-modal frame (like review services) is focused (so hitting 'process now' should show you the work)
  • fixed a variety of old hacky wx close-window veto tests. the 'close client?' confirmation dialog will now reliably veto a close requent on 'no'/cancel, dialog close events that are vetoed (such as closing the manage tags dialog with pending tags) will now veto more than just the first time, and several bad media viewer archive/dupe filtering cancel and end-of-window events should now work more cleanly and correctly. users who had crashes at the end of filtering may find they are stable again
  • as a quick patch against some multiline notes and statuses, list controls now force single-line text in all cells
  • list controls now tooltip all cells
  • fixed the shutdown splash not updating after the daemons shut down (lmao)
  • 'modal' message dialogs, which are created by blocking maintenance tasks such as vacuum, will no longer raise the program to the foreground on creation
  • should have fixed the taglist vertical positioning jank that could occur in the row after a tag with a tall emoji unicode character (and also sometimes kanji/hangul)
  • fixed a typo bug that was throwing an error for the upnp port widget in the local client server management panel when 'allow non local connections' was checked
  • improved stability of bandwidth review panel bandwidth rules refresh
  • improved stability of review services bandwidth rules refresh
  • improved some dialog cleanup code
  • reverted a bad environment-setting change put in last week that was causing some running-from-source users trouble
  • misc qt code cleanup

  • the rest

  • updated the default pixiv tag search downloader to one submitted by a user. it now uses their api
  • updated the default twitter username lookup to a downloader submitted by a user. it fetches just the media tweet feed, making it more efficient. also added (but not linked by default) is a new tweet parser that can fetch most videos using a third-party site, advanced users may wish to play with this
  • added a {file_id} term for file export phrases that substitutes a unique and permanent numerical file identifier
  • fixed an issue where idle maintenance jobs could sometimes sneak in a few milliseconds of work during certain long shut down pauses, such as while waiting for a 'should I do shutdown work?' dialog to return. program shutdown should be snappier for many users as forced startup delays in these calls will no longer trigger
  • added a date 'encode' string transformation rule, which takes an integer timestamp and converts it to a pretty date string. the date rules are now renamed to the clearer 'datestring to timestamp' and vice versa
  • fixed page parser edit panel's 'test parse' when string transformations perform pre-parsing conversion. the handling and passing of test data for all the panels here is cleaned up throughout
  • system:limit predicate edit panel now has a small label describing its sampling behaviour
  • updated the various 8chan links in the client and help to 8kun, let me know if I missed any, and added Endchan bunker link to help menu
  • improved some misc status text handling across the program
  • refactored cache and manager code into different, simpler files
  • updated sqlite on windows build to 3.30.1

• version 375

  • qt

  • disabled the failed legacy high dpi scaling mode experiment (which was scaling up thumbnails and media in an ugly way) and returned to font-size-based natural ui scaling as set by the OS. a couple of non-font things like bitmap buttons and various layout margins are too small on >100% UI scale, and the splash screen is borked again, but it looks clear again. I'll keep working on this
  • fixed the custom taglist at >100% UI scale, which was spacing its tags at the wrong text height. this should survive changing ui scale while the program is open and environments with multiple monitors at different ui scale
  • re-fixed a critical old media-viewer-close-on-video memory leak from wx code to qt code. this was also a cause for some child ffmpeg processes not being terminated
  • fixed the media viewer not redrawing correctly when the media size completely exceeds the canvas window size
  • fixed the loading of the shortcut edit panel when the shortcut set a tag
  • fixed some url class edit path component ui
  • fixed and cleaned up some 'safe window size/position' calculations that were missing out the total frame geometry, meaning some dialogs were not moving up and left enough to show entirely on screen, and dialogs with parent-dimension gravity were not calculating initial size accurately
  • fixed focusing on the already-open manage tags text input when you hit 'manage tags' on a canvas with a manage tags dialog already open
  • fixed the html formula rule edit ui actually rendering html tag labels, lmao
  • updated boot-password entry to use the normal hydrus text entry dialog, and fixed a hydrus password cancel not setting a 'clean' exit for the next boot
  • fixed page layout splitter sash positions not resetting nicely from the menu command
  • fixed keyboard delete in the manage urls dialog
  • popup message titles are now in bold
  • popup message titles should now multiline correctly and fill available width
  • the popup messages manager should now set its min/fixed width more sensibly
  • subscription popups now will be wider if space is available
  • wrote a new class to manage better asynchronous updates for future Qt ui presentation
  • the file, pages, and pending menubar menus, which all require a db hit to generate, now operate on this new update class. all three should update faster when able and more politely and smoothly wait when the db is busy
  • reduced some accidental blocking in an old ui-update routine that kicked in when it was running hard
  • if the media_viewer frame type is set not to remember its 'last size', it will now instantiate with a small min size
  • when pasting new queries into a sub, if there are more than 5 or 50 that are already in or new, they will be rendered in a more compact way in order to stop the notification dialog growing too tall
  • improved stability of page update, splash screen update, and perhaps pubsub update

  • new file maintenance jobs

  • added a new 'check for missing files' file maintenance job, where if the file is missing and has urls, those urls will be queued up in a new url downloader for redownload. the file record is not removed, preserving archive/inbox and import time
  • added a new 'check for invalid files' file maintenance job that does the same deal as above with an additional expensive byte-for-byte content check if the file is not missing
  • added a new 'check for invalid files' file maintenance job that only cares about invalidity--if the file is present and invalid, it is moved out but the file record is not removed

  • the rest

  • network jobs that receive low-bandwidth error codes from the server now use a separate wait routine (previously, they piggybacked on the connection fail retry system). they have a separate cog-menu action to override these waits
  • the time delay multiple for connection errors and serverside bandwidth problems are now editable under options->connection. old default was 10 seconds base, now 15 and 60 seconds respectively
  • updated the danbooru login script
  • improved the precision of the thumbnail size estimate in database migration
  • the alphabetisation of a url class's GET paramaters on normalise is now optional. it is a new checkbox on the url class edit panel
  • when a default object fails to load from a png path, a simple error is now written to the log
  • misc cleanup

• version 374

  • qt environment/build

  • macOS build is useable! tab drag and drop position calculation doesn't work yet, so intra-client file DnDs and tab rearrange DnDs are disabled for now. borderless fullscreen is also disabled, feedback on this vs maximise would be appreciated
  • fixed a critical bug in the macOS release that was resulting in 100% CPU repaint loop for the canvas viewer when media was loaded (wew). this may have affected certain other platforms in some situations
  • the linux build has a variety of common library files removed, letting your OS rely on higher compatibility system defaults. this _should_ clean up font and other issues for users running on very new/old system libraries. if you cannot run 374, please let me know your distro and version and any error messages
  • the special linux running from source document is updated, including info about Arch and PyQt5
  • fixed a windows build issue that meant some animated gifs were not able to load and render correctly
  • fixed a precise time fetching issue for users running from source with python 3.8
  • high dpi scaling should have improved support. please report on bad layout issues and other artifacts
  • fixed creating a serialised object png when using PyQt5
  • fixed file save dialogs with filetype filters when using PyQt5
  • fixed an important menubar related memory leak
  • _seem_ to have fixed an important media viewer memory leak

  • qt ui fixes

  • fixed pages not collecting and sorting on creation if they do not have to, which restores the 'preserve flat unsorted order' behaviour of session loads and file drag and drop page tab creations
  • fixed the cursor not unhiding on move in the media viewer when over an animation or static image
  • fixed the issue where a new thumbnail panel would double-up with the old one for half a second if a menu caused the panel swap
  • reworked the elided (text that cuts off...) label code to more reliably work on single lines, which fits our purposes. the network job control (esppecially on subscription popups) and top hover window should now show their long statuses without changing their parent panel's layout
  • updated a variety of old text-wrap-width wx-hacks texts to instead auto-fill available space
  • the various downloaders should now be careful about handling large status texts. if a multiline error or html page slips in to a status somewhere, your download pages' lists should no longer go nuts with very tall spam-filled status cells
  • hydrus->discord drag and drop should be fixed if the BUGFIX is on!
  • fixed page tab drag and drop to do live drag selection with 'do not follow' behaviour (this is switched by holding down shift during drag), and, in this case, got it to return to the original page's neighbour/parent once the drop is complete
  • fixed 'center' dialogs positioning on the center of their parent windows, rather than the center of the primary screen
  • fixed the hover windows not passing shortcuts up to the media viewer when not consumed
  • fixed some misc 'can I consume a shortcut' focus/active checking code
  • fixed the various hide/parents/siblings tag menu items for tags with counts
  • fixed the main gui and other non-dialog windows remembering their pre-maximise/fullscreen sizes if set to remember size and previously closing while maximised/fullscreened
  • menubar menus should now show description text in the main gui statusbar on mouseover of their items
  • fixed a bad menu initialisation in the canvas preview panel
  • fixed a little page splitter bork and improved size of preview window on initial boot
  • fixed the edit notes dialog when launched from the media viewer
  • fixed a couple of text edit issues in edit url class panel
  • fixed page up/down scroll for taglists
  • fixed page down scroll for thumbnail grid, and fixed page up/down distance
  • fixed thumbnails not scrolling into view if they are keyboard-selected slightly off screen but within the scroll option percentage threshold
  • misc layout and style cleanup
  • misc refactoring

  • misc

  • you can now set the maximum size of duplicate filter pair batches (default 250) under options->duplicates
  • when an ipfs service fails to pin a file and returns no hash or the empty multihash, this is now recognised, info dumped to log, a simple popup message sent, and the job continued. this is just a patch--better error handling here will come later
  • if the client or server are launched with a custom temp_dir that does not exist, it will now attempt to create it (previously errored out)
  • fixed a clean exit after certain client boot fail error handling, and repeated cleaner exit for the server
  • added some new memory profiling actions to the help->debug menu
  • parallel subscriptions should now initialise with less of an aggresive CPU spike
  • if the client or server crash before the application can be launched, the crash log is now called hydrus_crash.log. if the db dir is not yet established, it will now try to find and put it in your desktop and, failing that, then your user dir
  • the client no longer prints 'booting db' twice
  • a variety of misc code cleanup and fixes

• version 373

  • qt

  • hydrus now uses Qt for its client's user interface, migrating from wx. this is thanks to a huge effort by a user, who delivered converted code for hydrus dev to finish off
  • a number of hacks and patches remain to compensate for old systems, which hydrus dev will slowly clean up in normal work. ui bug and layout issue reports would be greatly appreciated
  • shortcut storage had to be converted from fixed wx enums to an independant system. there is a small chance that one of your shortcuts, particularly if it is on the numpad, may have been converted wrong (unusual Enter/Return buttons may be hit here). if one is not working, please check what hydrus thinks it is and try re-entering it
  • added tentative support for 'Mode_switch' keyboard modifier, for X11 users (and perhaps some users' AltGr?)
  • autocomplete results can now float in a popup window in dialogs like manage tags! they'll still embed by default, but there are now separate float/embed options for 'main gui' and 'other frame' a/cs
  • autocomplete results can now float in linux and macOS ok!
  • page drag and drop now navigates as you drag, so dropping into a page of pages works by you hovering over it and then dropping in the tabbar below, inserting exactly where you want the page to be
  • a couple of text inputs in the program--the watcher and gallery search pages' text inputs, particularly--now use nicer 'placeholder' text, which isn't real and only shows as grey text when the input is empty
  • for now, moved to icons for thumbnail 'has audio/duration' indicators, rather than the custom labels
  • to run the hydrus client from source, qtpy is now needed. either pyside2 (default) or qtpy5 is needed. QtCharts is optional. wx and matplotlib are no longer needed

  • misc

  • 'archive/delete filter' now appears even when no file is focused. it also appears when no files are selected--and will apply to everything
  • the system predicate edit panels now support static buttons for easy one-click select for common predicates. duration, has audio, limit, and num tags now have these
  • system:duration and system:num tags now render a special label if they are >0 or =0
  • system:untagged is now removed from the normal list
  • fixed a critical cpu inefficiency in the file maintenance manager's new always-on maintenance, which was lagging several users' browsing sessions while it was working
  • fixed ctrl+mousewheel tag autocomplete results navigation to skip over multirow parent results
  • fixed an issue where resetting to default bandwidth rules for a network context would not update the ui properly
  • fixed a bug when adding a parent/sibling from autocomplete results list
  • the serialised png export folder now catches when a manually inputted export path's directory does not exist
  • reduced metadata update lag of pages with very large media collection groups
  • the inaccurate 'add tags based on filename' button is now called 'import with tags'
  • fixed a database UNIQUE issue when two duplicate gui session save calls happen within one second
  • the server's lock_off command now works with the Hydrus-Key header auth (rather than hanging indefinitely wew)
  • the server now caches hashed access keys in the session manager, in memory, to avoid a db hit on access-key based reauthentication, and in instances where this authentication requires a db hit, now cleanly provides an appropriate 'serverbusy' error
  • improved some media object memory management and speedy cleanup
  • improved boot fail graceful exit
  • removed a bunch of defunct flash (swf) hacks from media viewer code
  • bunch of misc non-qt cleanup as I went through the code
  • fixed a bug with rendering network credentials for human display
  • cleared out the ancient tag archive sync advanced help and added a stub for the new tag migration window
  • various help updates around wx->Qt

• version 372

  • petitions processing page

  • the selection taglist now displays the raw 'storage' tag view, before siblings are applied
  • added a noneable spinctrl to control how many files are shown on a petition row double-click. it samples randomly and defaults to 256
  • I think I fixed the issue where the petitions taglist sometimes hangs on to some old tags after a petition process event

  • the rest

  • you can now customise the animation scanbar height and nub width under options->media
  • all users now see the number of open pages in the pages menu
  • added approx total session 'weight' to the pages menu. this is an early test and will do more and update more frequently in future
  • added add/remove tag to favourites to the taglist right-click menu
  • collapsed the taglist right-click menu a little, as it was getting a bit tall
  • added https://gitgud.io/koto/hydrus-archive-delete, a web browser archive/delete filter, to client api help
  • added clipboard import/export buttons to the edit tag filter panel for the new favourites
  • added 'open in a new page' to media viewer right-click menus, just to put the current single media in a new page
  • fixed the url class edit panel not initialising with the new referral options correctly
  • the call that publishes new subscription/import folder media to pages now does so more politely to the gui when the db is busy
  • subscriptions will no longer start if global network traffice is paused
  • the 'hard-replace siblings/parents' action under manage tags is now a local-only operation for tag repositories. clients with unusual sibling and parent application will no longer affect the repos they sync with
  • on program shutdown, if a daemon takes more than thirty seconds to shutdown (which can happen in odd situations, like if a subscription run is paused by global network traffic pause, leading to shutdown deadlock), the client will stop waiting and continue with other shutdown tasks
  • fixed an error with client 'already running' fast exit
  • fixed a different error with server 'already running' exit choice
  • updated some ffmpeg calls to fix certain OS problems
  • fixed a help link to todolist recommendation and added link to new ptr guidelines

• version 371

  • the edit tag filter panel now has load/save/delete buttons at the top to manage tag filter favourites. it starts with a handful of examples
  • sorting thumbnails by num tags or namespaces now uses the 'single' tag display context
  • the 'sort by media views/viewtime' sorts now do not put the other (viewtime/views) as an implicit secondary sort, so as to better let the user's secondary sort be used
  • highlighting a downloader should now not be able to create a page with duplicate thumbnails
  • all thumbnail pages now do an addition de-dupe check when they are created with media
  • when a gallery page parser now adds new urls to a file import list, urls that are invalid will now be skipped (previously, they threw an error and failed the parse
  • fixed a bug where if a default collect is set, pages without a collect (e.g. download pages) would nonetheless initialise with collected+sorted initial media on session load
  • file imports now publish the same 'refresh existing media metadata' call as the file maintenance system, meaning if the import already exists in the gui session as an 'unknown thumb', it should now refresh itself correctly
  • if the media canvas is called to display an invalid media (due to mime mixup or a faulty parse that slips through), it should now better recognise that and skip/dump out
  • fixed import of videos that have 'Duration:' in their title metadata
  • improved the error reporting when the old options object fails to save
  • removed some old ratings dialog position options storage that was causing errors on certain ratings dialog ok events
  • url classes now support options regarding the 'referer' http header they send (their referral url). you can set an optional converter to generate a referral url based on the url class's url and choose to always use the given referrer if available, never use a referrer, use the converter if no referrer is available, or always use the converter
  • the network report mode now reports on referral urls used in requests
  • the 'quoted' referral url (a unicode workaround) is now only applied if the referral url cannot be encoded to latin-1
  • the janitorial petitions processing page now lets you copy tags and left/right tags of pairs with a right-click on selected checkbox rows
  • cleaned a little server code
  • improved how the server sets and releases its 'currently busy' mode
  • the server no longer does <5min vacuums in a backup command
  • added a specific 'vacuum' server POST command that forces a full vacuum
  • added 'lock_on' and 'lock_off' server POST commands to lock the server and shut down the db, and restart
  • the new vacuum, lock_on, lock_off, and a 'is server busy?' check commands are added to the services->admin menu
  • added 'pause and disconnect' ability to the database mainloop
  • added some unit tests for url classes and the new referral url conversions and server commands
  • cleaned some of the thumbnail banner/icon drawing code
  • some misc label fixes

• version 370

  • tag display updates

  • the old tag censorship system is now replaced by a broader tag display manager that will deal with tag storage and presentation settings. this is the first step, and some of it is not yet completely functional or as efficient as intended
  • management for the new tag display manager can be found under services->manage tag display. you can set per-service and all-services filters for 'multiple' display like the 'tag selection' boxes and for 'single' file views, like thumbnails and the media viewer
  • existing censorship rules will be added to the new manager and will apply to 'selection list' and 'single media' display rules
  • censorship/display rules no longer apply to underlying storage views, which are unfilterable for now, so the manage tags dialog and the autocomplete lookup will now show all tags
  • page and media viewer taglists now have new right-click menu options for hiding--they will provide hide options for the specific tag clicked, and its namespace more broadly, and will apply immediately to that kind of taglist (previously, this was just the tag, and launched the tag censorship edit panel)
  • all 'tag manager' objects behind every media object in the client now pre-compute cache layers for different tag presentation contexts. operations such as sibling collapse are now only done on file load or new siblings
  • for now, initial media load will take slightly longer, but various tag display updates and autocomplete tag fetches on media will be faster
  • changes to siblings and the new tag display rules will now trigger a reliable (although, for siblings, delayed by a few seconds) and complete tag list and thumbnail and media viewer refresh
  • changes to tag presentation will now correctly update collection thumbnails
  • some complicated sibling display and counts are now more precise
  • cleaned up some tag/siblings/thumb refresh notification code
  • cleaned up all tag manager access code
  • cleaned up a variety of related tag fetching, counting, and display code

  • the rest

  • added 'system:modified date'. it works just like system:time imported
  • files with duration but no audio will now have a ' ▶ ' label in the top-left of their thumbnails, like the 'has audio' one. you can edit this label under options->audio and duration. I don't really like how this looks, so maybe we'll go to icons. let me know what you think
  • fixed an iteration timing bug in the new asynchronous repository mappings processing that meant large lists of mappings within an update object may be occasionally truncated, leaving some mappings unprocessed. this would more affect users on slower machines running 'process now'
  • any tag repository content updates issued in the last eight weeks will be scheduled for reprocessing to cover the above issue and fill in gaps for most user situations. since the vast majority of the data was added as intended, they should catch up very fast
  • added two pixiv url classes for their new url format
  • the edit subs panel now recommends users break up subs with >200k urls
  • the 'separate' subscriptions button now has a new 'break in half' option. subscriptions that have more than 100 queries will auto-choose this to separate
  • the 'quality info' button for advanced users' edit subscription panels now gives the option to additionally copy the info to your clipboard in CSV format
  • the lists on the gallery downloader, thread watcher, subscription, and subscriptions panels now sort their progress column by ( y, x ) (given a total status of x/y). previously, this was preceded by a percent-done sort
  • the hydrus network engine now recognises 429 bandwidth responses
  • on 429 or 509 bandwidth responses, network jobs will now go through the regular reconnection delay loop and try again later (previously, they just failed)
  • added 'tag migration' to 'services' menu for quick-launch
  • tag migration's 'go' action now skips the second confirmation if you are in advanced mode
  • expanded the 'reset' review services button (only visible to advanced users) to allow 'softer' resets that simply reprocess definition/content without deletion (a 'filling in the gaps' command)
  • fixed the 'process now' review services button disable check, which was being overzealous
  • cleaned up some of the new repo 'caught up' checking code
  • improved stability of review services 'refresh account' call
  • the client api /manage_pages/get_page_info call now returns a list of hash_ids beside the list of hashes, in simple or not simple mode
  • fixed a bug where tag import options that still had a secret deleted service reference were causing tag-parse errors on import jobs
  • fixed some other places that were not handling service disappearance neatly
  • added a note to the install/backup help to mention not to use continuous cloud-sync backups on your live db directory
  • misc unit test refactoring

• version 369

  • file maintenance

  • the file maintenance manager now works continuously in the background, optionally in both idle and active time, with two different throttles, which are now always active
  • as usual, the default throttles are low-load (1 heavy job every 2 (idle) or 20 (normal) seconds), so as not to interfere with your browsing or other programs--feel free to speed them up as you wish
  • the options for file maintenance under 'maintenance and processing' are updated, and quick-pause actions are now available under database->maintain->file maintenance
  • the file maintenance manager no longer works on shutdown
  • the file maintenance manager will now only make a popup if it is started by the user--it otherwise now works silently in the background
  • the file maintenance manager now weights its jobs, so quick jobs will run faster and heavy jobs will space out more. exact weights, if you are interested, are now under the 'see description' button on the maintenance panel
  • file maintenance jobs now report to the debug file report mode
  • improved some misc file maintenance code, particularly with how the panel talks to the manager
  • media with new metadata will now refresh their thumbnails (for now, this means updating the has_audio icon)

  • modified timestamps

  • the client now records file modified timestamps of all file imports!
  • on update, the retroactive population of this data for all existing local files will be scheduled on the file maintenance system, which has a new job type for this
  • the modified time now appears on a file's information lines that present on a right-click
  • the modified time can be sorted with the new 'file: modified time' sort

  • the rest

  • added lexicographic sort by subtag (ignoring namespace) to the normal taglist sort selection
  • reworded the sort by lexicogrphic (grouped by namespace), to be (group unnamespaced)
  • the export files panel now has an explicit button to change the neighbouring .txt file tag services
  • on duplicate merge action options panel, 'sync archive' is no longer disabled for advanced users' 'alternates' duplicate action
  • split the download and process sync components of repositories a little
  • added a 'download now' button to repositories' review services panels, to hurry up metadata/update download when possible
  • the 'process now' button's enable/disable states should now be more reliable
  • the 'refresh account' button now disables when a repository is paused
  • improved stability of 'process now' button post-job updating
  • added a subscription option to the downloading option panel to change how many file-fails in a run will cause a sync to stop working early
  • re-added the truncated image loading mode to the debug->data actions menu. this has hung indefinitely with some bad files, so it not on by default
  • fixed an issue with copying an external local booru url with a upnp port
  • fixed an unrecoverable ui hang when a modal popup wants to self-terminate while a child yesno is open
  • if on a hydrus request the session key is invalid (due, for instance, to a recent serverside session clearing :^)), the session key cookie will now correctly be cleared clientside so a new one can be generated automatically on the next request
  • hydrus services can now take the access key as their credential using the 'Hydrus-Key' header. more options will come here, basically the same as the client api
  • network jobs waiting on a login process now continue faster once the login is complete (5s sleep cycle down to 1s)
  • perhaps fixed some linux problems with tag migration panel, perhaps not
  • caught and silenced a rare unimportant services shutdown error
  • updated to opencv 4.1.1 on the linux build
  • updated windows ffmpeg to 4.2.1

• version 368

  • multiple local tag services

  • you can now add additional local tag services under services->manage services!
  • new local tag services will appear in manage tags and tag import options and so on, just like when you add a tag repository
  • you can also delete local tag services, but you must have at least one
  • the default local tag service created for a new client is now renamed from 'local tags' to 'my tags'. any existing user with their local tag service called 'local tags' will be renamed on update to 'my tags'

  • ptr migration

  • the ptr has been successfully migrated to user management! hydrus dev is no longer involved in running or administering it. the old bandwidth limits are removed! it has the same port and access key, but instead of hydrus.no-ip.org, it is now at ptr.hydrus.network
  • on update, if you sync with the ptr, you will get a yes/no asking if you want to continue using it at the new location. on yes, it'll update your server's address automatically. on no, it'll leave it as-is and pause it. if you still have a connection to my old read-only file repo, that will be paused
  • changed the auto repo setup command to be _help->add the public tag repository_. it points to the new location
  • as repo processing and related maintenance is now nicer, and secondarily since bandwidth limits are less a problem for the ptr specifically, the default clientside hydrus bandwidth limit of 64MB/day is lifted to 512MB/day. any users who are still on the old default will be updated
  • updated the help regarding the public tag repository, both in general description and the specific setup details
  • a copy of the same sanitized and frozen PTR db used to start the new PTR, and convenient tag archives of its content, are now available at https://mega.nz/#F!w7REiS7a!bTKhQvZP48Fpo-zj5MAlhQ

  • the rest

  • fixed a small bug related to the new 'caught up' repository mechanic for clients that only just added (or desynced) a repository
  • rewrote the tag migration startup job to handle specific 'x files' jobs better--they should now start relatively instantly, no matter the size of the tag service
  • on 'all known files' tag migrations, a startup optimisation will now be applied if the tag service is huge
  • fixed the tag filter's advanced panel's 'add' buttons, which were not hooked up correctly
  • the internal backup job now leaves a non-auto-removing 'backup complete!' message when finished
  • on update, server hydrus repositories will collapse all their existing content timestamps to a single value per update. also, all future content uploads will collapse similarly, meaning all update content has the same timestamp. this adds a further layer of anonymity and is a mid-step towards future serverside db compaction (I think I can ultimately reduce server.mappings.db filesize by ~33%). if you have a tag repo with 10M+ mappings, this will take some time
  • hydrus servers now generate new cert/key files on boot if they are missing. whenever they generate a new cert/key, they now print a notification to the log
  • misc help fixes and updates, and removed some ancient help that referred to old systems
  • corrected journalling->journaling typo for the new experimental launch parameter

• version 367

  • tag migration

  • added htpa and tag service sources for parents/siblings migration that support filtering for the left and right tag of each pair
  • added htpa and tag service destinations for parents/siblings migration
  • added unit tests for all parent/siblings migration scenarios
  • misc improvements to mappings migration code
  • reworded some of the tooltip/tag filter message text to more clearly explain how the filter applies to migrations
  • the tag filter edit panel now has a 'test' area where you can put in an example tag to see if it passes or is blocked by the current filter

  • the rest

  • fixed an issue with auto-no-ing yes/no dialogs throwing errors on exit. I am sorry for the inconvenience!
  • thumbnails now show the 'has audio' string on their thumbnails
  • 'sort by file: has audio' added!
  • icons drawn on thumbnails are now adjusted to sit inside the border
  • added increment/decrement numerical ratings actions for media shortcuts! if a file hit by this action has no rating, it will initialise with 0/1 stars or max stars. please forgive the ugly expanding ui in the shortcuts panel here--I'll rewrite this to layout more dynamically in future
  • client repository services now track whether they are 'caught up' to their repos, which for now means processed up until at least two weeks ago, and will prohibit uploading new content until the client is caught up
  • repository review services panels will now display the 'caught up' status below the 'processed' progress gauge
  • repository review services panels will no longer duplicate 'account' status problems in the 'this client's network use' status line--both lines now refer to service/account functionality separately
  • repositories will now put in 'unknown error' when an empty error reason slips through the 'no requests until x time' reporting process
  • the new thumbnail and media viewer right-click menus now collapse the selection info lines at the top to just the top line and places all the rest (and in complicated file domain situations, this can be a long list) in a submenu off that line
  • the new thumbnail 'remove' submenu has separators after 'selected' and 'all' to reduce misclicks
  • reworded a couple of things in the manage shortcuts panel to be more clear
  • added wildcard support ('eva*lion') and namespace wildcards (like 'character:*') to the advanced OR text input parsing
  • fixed a rare issue with the duplicate filter being unable to go back or retreat from an interstitial confirm/forget/cancel dialog when every pair in the current batch cannot be displayed (such as if at least one of the pair has been physically deleted). the filter now catches this situation, informs the user, and closes itself gracefully
  • added two extremely advanced and dangerous launch parameters for database access testing
  • couple of misc fixes and cleanup

• version 366

  • tag migration

  • wrote a unified mass-migrate pipeline to make moving large amounts of data in and out of the client more powerful and more pleasant like other recent non-interrupting changes
  • the advanced content update dialog is now renamed to 'migrate tags', both on the review service panel and the manage tags dialog, and has been completely revamped to reflect the new migration pipeline, which works on a location-agnostic (content type, source, filtering, destination, action) model
  • the advanced content update dialog is now a non-modal frame--you can keep using the client while it is open, and it will not prohibit the popup from appearing while it works
  • added hta and tag service mapping sources to the new migration pipeline, with hash conversion, content status filtering, file domain filtering, specific hash filtering, and tag filtering
  • added mappings clear deletion record action to the normal content update pipeline
  • added hta and tag service mapping destinations to the new migration pipeline, with appropriate choosable content action (add for HTAs, add, delete, clear deletion record for local tags, and pend, petition for tag repositories)
  • added stub list sources and destinations for testing the new migration pipeline
  • wrote comprehensive unit tests to test mappings migrations with hta source and destination, including tests for file domain, hash type conversion, and tag filtering
  • wrote comprehensive unit tests to test mappings migrations with local tags or tag repository service source and destination, including tests for file domain, hash type conversion, tag filtering, and content status
  • adjusted the tag archives to have an optimise call separate from the commit call, so you can do several big jobs in a row faster, or pre-optimise, or top up a well-optimised db without wasting time re-optimising every commit
  • added a Close method to Tag Archives just for a nicer explicit exit
  • wrote a HydrusTagPairArchive to store sibling and parent tag pairs
  • deleted old tag db migration and archive syncing code, which is no longer used
  • deleted some related old hta import/export code
  • removed the 'tags' export option from the thumbnail share->export menu
  • accessing 'migrate tags' is no longer gated by advanced mode on the manage tags dialog cog menu
  • the hydrus database now creates a name.temp.db database while it is running to support long-term temp jobs such as the new migration. this file is otherwise unimportant and is deleted on a clean exit

  • thumbnail menu rework

  • shifted the thumbnail menu around to group 'view' vs 'action' commands together more sensibly and bury less frequent commands away from the top list
  • the 'remove' menu command is now a submenu with filters for 'selected/not selected/ all/inbox/archive'. it also displays on a right-click with no focused file
  • 'delete' now has separator on both sides to reduce accidental clicks
  • 'share->open' is now moved up a level and inherits 'open in new page' and 'open externally'
  • the 'remote services', 'file relationships' and 'regenerate' submenus are moved to the 'manage' submenu
  • made similar changes to the media viewer menus, including grouping the zoom/fullscreen/slideshow commands together and making the zoom commands a submenu
  • misc wording and count changes to this menu

  • the rest

  • added 'main_gui' shortcuts for 'refresh_all_pages' and 'refresh_page_of_pages'. the second shortcut refreshes all pages under the most immediate page of pages parent
  • fixed an issue where under-construction OR predicates were not displaying with the system predicate list
  • the 'import local files' frame, which pops up when you drop some paths on the client, is now on the new panel system and sizes and cleans itself up more sensibly as a result
  • refactored the remaining 100-odd yesno dialogs to the one that works on the new panel system
  • misc yesno dialog message and logic cleanup
  • fixed an issue where the export files dialog could hang indefinitely if the filename phrase involved long shared tags that resulted in duplicate paths for the first 245 characters in length. now, the long-filename truncation is done before the de-duping ' (n)' text is appended, and the length limit is reduced to 240
  • removed the now-defunct 'regen similar files metadata' command under the database menu--this is now handled in the new file maintenance processing system
  • improved database optimisation code to better check if empty/small tables have suddenly grown and improved quality of optimisation data for frequently emptying tables
  • fixed the 'clear orphan file records' maintenance task, which was not performing the final clear on the correct domain
  • fixed some bad error presentation
  • fixed the duplicate page not refreshing maintenance numbers after maintenance job completion
  • cleared out a bunch of old py2to3 safety code, maybe sped up some sibling/parent/mappings stuff
  • misc ui cleanup and small fixes

• version 365

  • new repo processing

  • improved the new asynchronous repository processing system in several ways

  • - it now uses the time it is allotted more accurately. when it has 0.45s to work, it hits this mark more often, especially on slower machines
  • - it is now politer to the ui if plenty of other work is going on--if the db is loading search results or you are viewing video, it should pause for milliseconds as needed
  • - it can now work up to 90% of the time during a manual 'process now' run
  • - when it is working faster than expected, it accelerates its load to operate more efficiently
  • as a result, the new system should now have faster rows/s and lag out the ui less

  • client api

  • improved how parameters are fetched and tested against expected type and given default values if appropriate, and updated all client api code to use this new system
  • added /manage_pages/get_page_info, which gives simple or detailed info on a given page, found by page_key
  • added page info response to hdd importers, simple downloaders, url downloaders, gallery downloaders, watcher downloaders--they say their pause status and file/gallery import info as appropriate
  • added page info response to file import caches--they say their status and num_done/num_to_do, and in detailed mode report file import item info, which is url/path, created, modified, and source times, current status, and any note
  • added page info response to gallery logs--they say their status and num_done/num_to_do, and in detailed mode report gallery log info, which is url, created and modified times, current status, and any note
  • added page info response to thumbnail panes--they say their total num files, and in detailed mode list their ordered hashes
  • started some help for this expansion, but it will need some feedback and more work to finish
  • the client api now sorts /get-files/search_files results by import time, newest to oldest. this first hardcoded sort comes to help implement booru-like pagination, but will be expanded to support more types as I flesh out the ui side (as below) as well
  • hydrus services, including the client, should now be able to handle larger request header+path total size (16KB->1MB). this helps some larger GET queries in the client api. let's see how this goes
  • client api is now version 11

  • collect improvements

  • the collect data attached to pages is updated to its own object. the default value and existing page settings should update. all ui now handles the new clean object, rather than the old messy list
  • the new collect object supports an option for whether to collect 'unmatched' thumbs or to leave them separate. this displays in the ui as a dropdown beside the collect-by checkboxlist
  • to better distinguish between unmatched singles and matched collections with just one item, all one-item collections will now act as collections, with the little '1' collection icon in their corner (previously, they were split into singles). if this is annoying, I will add another option to control whether this occurs
  • removed some old 'integrate media into existing collected structure code' that was complicated, never used, and now broken
  • misc sort/collect refactoring
  • deleted some old unused collection code

  • the rest

  • entering tags in the filename tagging panel, either for all or just selected, now pushes those tags to the 'recent tags' list in later manage tags dialogs
  • added a framework to start sorting search results before the system:limit is applied--I will soon extend this to start catching the current ui sort (say, 'largest files first', and cut a system:limit appropriately, rather than the current random sample)
  • added a faster table size check on the analyze maintenance call that will recognise fast-growing tables (e.g. initially empty/tiny repository processing tables that may have seen a ton of recent work) and schedule them better (this was previously potentially hanging certain maintenance checks/shutdown by several minutes when hitting a surprisingly giant table)
  • reduced the analyze frequency for established tables
  • the client will now explicitly count as 'idle' and 'very idle' during shutdown maintenance time, in case any shutdown job is considering that for how greedy it should be with work time
  • fixed an issue where appending new media (thumbnails) to a page that already had that media but within a collection could create a duplicate media entry and invalidate some internal data links to the old media
  • subscriptions will no longer print full traceback information when a network error causes a sync fail
  • updated to yet another deviant art file page parser. title tags and embedded image links should be fixed again, post/source time is not fixed
  • the deviant current art login script is confirmed to work for some users. my guess is certain people are getting cloudflare blocked or aren't being shown the new login page all the time yet, please send in any more info you discover
  • the client will now recover from a missing options object by putting in a fresh one with default values, including a popup notifying you of the error and giving you a chance to bail out
  • added a warning and link to the quicksync to the access_keys help page
  • if the os commands the client to close due to a log off or system shut down, the client will kindly ask for a bit more time do to so if it is available
  • updated the WTFPL license to v3

• version 364

  • repo processing makeover

  • repository processing is now no longer a monolithic atomic database job! it now loads update files at a 'higher' level and streams packets of work to the database without occupying it continuously! hence, repository processing no longer creates a 'modal' popup that blocks the client--you can keep browsing while it works, and it won't hang up the client!
  • this new system runs on some different timings. in this first version, it will have lower rows/s in some situations and higher in others. please send me feedback if your processing is running significantly slower than before and I will tweak how this new routine decides to work and take breaks
  • multiple repos can now sync at once, ha ha
  • shutdown repository processing now states the name of the service being processed and x/y update process in the exit splash screen
  • the process that runs after repository processing that re-syncs all the open thumbnails' tags now works regardless of the number of thumbnails open and works asynchronously, streaming new tag managers in a way that will not block the main thread
  • 'process now' button on review services is now available to all users and has a reworded warning text
  • the 1 hour limit on a repo processing job is now gone
  • pre-processing disk cache population is tentatively gone--let's see how it goes
  • the 10s db transaction time is raised to 30s. this speed some things up, including the new repo processing, but if a crash occurs, hydrus may now lose up to 30s of changes before the crash

  • the rest

  • users in advanced mode now have a 'OR' button on their serch autocomplete input dropdown panels. this button opens a new panel that plugs into prkc's neat raw-text -> CNF parser, which allows you to enter raw-text searches such as '( blue eyes and blonde hair ) or ( green eyes and red hair )' into hydrus
  • fixed the silent audio track detection code, which was handling a data type incorrectly
  • improved the silent audio track detection code to handle another type of silence, thank you to the users who submitted examples--please send more false positives if you find them
  • fixed an issue where thumbnails that underwent a file metadata regeneration were not appearing to receive content updates (such as archive, or new tags/ratings) until a subsequent reload showed they had happened silently. this is a long-time bug, but the big whack of files added to the files maintenance system last week revealed it
  • the 'pause ui update cycles while main gui is minimised' change from last week now works on a per-frame basis. if the main gui is minimised, media viewers that are up will still run videos and so on, and vice versa
  • a few more ui events (e.g. statusbar & menubar updates) no longer occur while the client is minimised
  • duplicate processing pages will now only initialise and refresh their maintenance and dupe count numbers while they are the current page. this should speed up session load for heavy users and those with multiple duplicate pages open
  • gave the new autocomplete 'should broadcast the current text' tests another pass--it should be more reliable now broadcasting 'blue eyes' in the up-to-200ms window where the stub/full results for, say, 'blue ey' are still in
  • fixed an accidental logical error that meant 'character:'-style autocomplete queries could do a search and give some odd results, rather than just 'character:*anything*'. a similar check is added to the 'write' autocomplete
  • fixed an issue with autocomplete not clearing its list properly, defaulting back to the last cached results, when it wants to fetch system preds but cannot due to a busy db
  • fixed GET-argument gallery searches for search texts that include '&', '=', '/', or '?' (think 'panty_&_stocking_with_garterbelt')
  • removed the pixiv login script from the defaults--apparently they have added a captcha, so using Hydrus Companion with the Client API is now your best bet
  • the client's petition processing page will now prefer to fetch the same petition type as the last completed job, rather than always going for the top type with non-zero count
  • the client's petition processing page now has options to sort parent or sibling petitions by the left side or right--and it preserves check status!
  • the client's petition processing page now sorts tags by namespace first, then subtag
  • the client now starts, restarts, and stops port-hosted services using the same new technique as the server, increasing reliability and waiting more correctly for previous services to stop and so on
  • the client now explicitly commands its services to shut down on application close. a rare issue could sometimes leave the process alive because of a client api still hanging on to an old connection and having trouble with the shut-down db
  • the file maintenance manager will no longer spam to log during shutdown maintenance
  • sketched out first skeleton of the new unified global maintenance manager
  • improved some post-boot-error shutdown handling that was also doing tiny late errors on server 'stop' command
  • added endchan bunker links to contact pages and github readme
  • updated to ffmpeg 4.2 on windows

• version 363

  • has audio

  • wrote a detection routine that can determine if a video has audio. it reads actual audio data and should be able to detect videos with a 'fake' silent audio track and consider them as not having audio
  • extended the client database, file import pipeline, and file metadata object to track the new has_audio value
  • flash files and audio files (like mp3) are considered to always have audio
  • all 'maybe' audio files (atm this means video) are queued up for a file metadata reparse in the files maintenance manager. your existing videos will start off as not having audio, but once they are rescanned, they will get it. this is one of the first big jobs of the new maintenance system, and I expect it will need some different throttling rules to finish this job in reasonable time--by default it does 100 files a day, but if you have 50,000 videos, that's a long time!
  • files now show if they have audio in their info string that appears on thumbnail right-click or the top of the media viewer. it defaults to a unicode character, but can be edited under the new 'sound' options page
  • added a system:has audio predicate to search for files with/without audio
  • updated file import unit tests to check 'has audio' parsing, and added tests for system:has audio

  • client api

  • the /get_files/file_metadata call now provides has_audio info
  • the /get_files/file_metadata call now provides known_urls!
  • added 'cookie management' permission
  • added /manage_cookies/get_cookies to get current cookies by domain
  • added /manage_cookies/set_cookies to set or clear current cookies
  • added/updated unit tests for the above
  • updated help for the above
  • client api version is now 10

  • the rest

  • system:hash and system:similar to now accept multiple hashes! so, if you have 100 md5s, you can now search for them all at once
  • the thumbnail right-click->file relationships->find similar files now works for multiple selections!
  • when system:hash was just one hash, it would run before anything else and complete a search immediately on finding a match, but now it works like any other predicate, checking for file domain and ANDing with other predicates in the search
  • the 'complete' file maintenance regen job now only does file metadata, not a complete thumb regen. its name and labels are updated to reflect this, and any existing job in the system will get the separate thumb regen job
  • the file maintenance manager now has a couple of how-to sentences at the top, and a new 'see description' button will talk more about each job type
  • the login script testing system now uses a duplicate of the existing domain manager (rather than a fresh empty one), so it will inherit current http headers such as default User-Agent, the lacking of which was messing up some tests
  • fixed the login script testing system not showing downloaded data
  • subscriptions with multiple queries now publish the files they have imported as soon as each query has finished, rather than waiting for the whole sub to be done
  • subscriptions now publish the files they have imported to page/popup even if they have an error
  • added 9:16, 2:3, and 4:5 to the duplicate comparison statement system, for various vertical social media types
  • the autocomplete tag search 'read', which appears on places like search pages, should now more reliably accept the current entered text when there are no search results yet to show
  • the autocomplete tag search 'write', which appears on places like the manage tags dialog, should now correctly accept the input (including appropriate sibling-collapse) when you select a 'stub' result while other results are still loading, rather than broadcasting the exact current text
  • fixed the deviant art file page parser to get source time--however the login script may now be broken/unreliable
  • fixed a missing dialog import when deleting a string transformation
  • reduced the base network connection error reattempt time to 10s (from 60s). there will be more work here in future
  • network jobs that are waiting on a connection error now have a reattempt wait override option in their cog icon menus
  • the post-bad-shutdown 'open your default session or a blank page' dialog will now auto-choose to open your default session in 15 seconds
  • a variety of ui-update events will now not fire as long as the main gui is minimised. as well as saving a sliver of resources, I believe this may fix an issue where long-running subscriptions and other import pipelines could sometimes put the ui in an unrecoverable state due to too many thumb-fade etc... events when the currently focused page was receiving new files while the main gui was minimised
  • maybe fixed a rare problem with deleting old pages
  • cleaned some misc code

• version 362

  • duplicates work finished

  • updated the duplicates help text and screenshots to reflect the new system
  • duplicate files search tree rebalancing is now done automatically on the normal idle maintenance routine, and its over-technical UI is removed from the duplicates page
  • the duplicate filter's resolution comparison statement now specifies 480p, 720p, 1080p, and 4k resolutions and highlights resolutions with odd (i.e. non-even) numbers
  • if the files are of different resolution, a new 'ratio' comparison statement will now show if either have a nice ratio, with current list 1:1, 4:3, 5:4, 16:9, 21:9, 2.35:1
  • added a 'stop filtering' button to the duplicate hover frame
  • made the ill-fitting 'X' button on top hover frame a stop button and cleaned up some misc related ui layout
  • added a 'remove this file's potential pairs' command to the thumbnail file relationships menu
  • if in advanced mode, multiple thumbnail selection right-click menus' file relationships submenus will now offer mass remove/reset commands for the whole selection. available commands are: 'reset search', 'remove potentials', 'dissolve dupe groups', 'dissolve alt groups', 'remove false positives'

  • the rest

  • added link to https://gitgud.io/koto/hydrus-dd/ , a neat neural net tagging library that uses the DeepDanbooru model and has several ways of talking to hydrus, to the client api help
  • cleaned up a little of the ipfs file download code, mostly improving error/cancel states
  • rewrote some ancient file repository file download code, which ipfs was also using when commanded to download via a remote thumbnail middle-click. this code and its related popup is now cleaner, cancellable, and session-based rather than saving download records to the db (which caused a couple of edge-case annoyances for certain clients). I think it will need a bit more work, but it is much saner than it was previously
  • if you do not have the manage tags dialog set to add parents when you add tags, the autocomplete input will no longer expand parents in its results list
  • fixed an issue displaying the 'select a downloader' list when two GUGs have the same name
  • hitting apply on the manage parsers or url classes dialogs will now automatically do a 'try to link' action as under manage url class links
  • fixed (I think!) how the server services start, which was broken for some users in 361. furthermore, errors during initial service creation will now cancel the boot with a nice message, and the 'running ... ctrl+c' message will appear strictly after the services have started ok the first time, and services will shut down completely before the db is asked to stop
  • improved how the program recognises shutdowns right after boot errors, which should speed up clean shutdowns after certain bad server starts
  • the server will use an existing server.crt and server.key pair if they exist on db creation, and complain nicely if only one is present
  • the 'ensure file out of the similar files system' file maintenance job result will now automatically remove from/dissolve the file's duplicate group, if any, and clear out outstanding potential pairs
  • a system language path translation error that was occuring in some unusual filesystems when checking for free disk space before big jobs is now handled better
  • like repository processing, there is now a 1 hour hard limit on any individual import folder run
  • fixed an issue where if a gallery url fetch produced faulty urls, it could sometimes invalidate the whole page with an error rather than just the bad file url items
  • subscriptions will now stop a gallery-page-results-urls-add action early if that one page produces 100 previously seen before urls in a row. this _should_ fix the issue users were seeing with pixiv artist subs resyncing with much older urls that had previously been compacted out of the sub's cache
  • until we can get better asynch ui feedback for admin-level repository commands (like fetching/setting account types), they now override bandwidth rules and only try the connection once for quicker responses
  • misc code cleanup

• version 361

  • duplicates

  • the duplicate filter now compares the pixel content of static image pairs of the same resolution--if they have the exact same pixels, a comparison statement is added, and if one file is a png and the other not (i.e. the png is likely a useless clipboard copy), the statement notes this and a strong duplicate score is applied
  • added 'system:is/is not best file of its group' to search for file kings
  • renamed 'system:num duplicate relationships' to 'system:num file relationships'
  • wrapped the two file relationship system predicates into one 'system:file relationships' stub predicate that opens to a dialog with two pred panels
  • added a 'add potential pairs' command to the thumbnail right-click file relationships menu, which will force-queue files for the duplicates filter
  • the duplicate filter now ensures the two medias' zoom is locked so they have the same width through a transition. furthermore, their current dragged top-left position is pinned in the same location. this ensures files that have slightly different resolution ratios (especially when they are just a couple of pixels off) still remain reasonably comparable when switching back and forth
  • reworked and simplified how position/drag delta is handled in the media canvas to support the above
  • fixed the 'custom action' button on the duplicate filter, which had no 'delete neither' choice and whose 'forget it' button cancelled the whole custom operation, making it impossible to custom action without deleting something. I have added a 'delete neither' green-text button to the front, as the default action
  • mr bones now reports on your potential, duplicate, and alternates numbers

  • tag autocomplete

  • greatly sped up tag autocomplete search when fetching from a current media view (i.e. from thumbnails in the search page)--it had some CPU-inefficient testing/counting that mattered at high media/tag counts
  • greatly improved cancelability of tag autocomplete search when pulling from a current media view--this was resulting in high lag when typing fast with multi-thousand results
  • fixed the gui-level tag matching test to match namespaced search inputs with offset subtags (e.g. 'character:aran' now matches 'character:samus aran'), both for wildcard and specific namespaces
  • when typing an explicit wildcard tag search that does not end in a *, you will now be presented with two wildcard options--one with the implicit * suffix, one without
  • fixed 'write' tag autocomplete inputs (like in manage tags) being able to search for chunky 'namespace:*' explicit wildcard searches

  • the rest

  • fixed the ipfs nocopy path translation control saving rows for client file paths outside of the main install path for non-Windows, where it was forgetting on save
  • renamed 'system:size' to 'system:filesize'
  • sped up some system:inbox searches
  • disabled a PIL 'load truncated images' backup mode, which on the current version can seemingly lead to infinite load hangs
  • file report mode now prints info when it deletes/recycles a path, including stack traces
  • fixed a long-running and silent 'port already running' bug related to setting services on the server that was stopping successful service-set-restart from the client in many situations. 'port is already running' checks that conflict with other processes will now give an immediate error to the client without saving any changes
  • the server now prints to the log as it stops/starts/has started its services
  • improved how the server can report certain 500 errors
  • the 'critical service tag/file reference' repository processing error has been improved: rather than reset the whole repository, it now pauses the repo and resets processing status for just the repo's 'definition' update files (without deleting any existing entries, so they should ultimately reprocess super fast) and also schedules a complete integrity and metadata check for all updated files
  • keyboard interrupts from the console should now trigger a clean exit request for the client
  • polite and forced shutdown requests when logging off should now trigger a fast exit (i.e. no yes/no dialog, no shutdown maintenance, but otherwise session saved and so on) for the client. this fast exit is noted in the log
  • moved the tag and rating service listctrls in duplicate merge options panel to the new listctrl object
  • moved the manage regex favourites listctrl to the new object
  • updated a bunch of yes/no dialogs to the new panel system
  • deleted some old unused dialog code and related unit tests
  • fixed up deletion-and-reimport file location handling for lingering media objects, which were not correctly forgetting combined local file deletion record on the reimport
  • improved shutdown error handling during repo processing
  • deleted the mishimmie default downloader

• version 360

  • tag autocomplete

  • after various tag autocomplete async work, fetch timings get a complete overhaul this week. the intention is for a/c jobs to appear as fast as possible, with good ui feedback, without interrupting ui while they work. feedback on how this works IRL would be appreciated

  • there are now just two autocomplete options under options->speed and memory

  • - whether autocomplete results are ever fetched automatically, defaults to true
  • - the max number of characters in the input that will cause just exact results vs. full autocomplete results, defaults to 2, can be None
  • namespaces are no longer searched from an unnamespaced query ('char' no longer matches 'character:samus aran'). this proved too slow for real use, and remains better available with explicit namespace searches such as 'character:*' or 'char*:*'
  • the 'exact results' character limit now also applies to subtags of namespace searches! so, entering 'character:a' will deliver the same short exact match results as just 'a'--no more gigantic lists when you put in a simple namespace
  • improved tag results caching to deal with the new non-namespace matching on subtag input
  • tag autocomplete dropdowns will now display a non-selectable 'loading results...' label when results take more than 200ms to load.
  • tag autocomplete dropdowns will now also display 'static' tags, such as 'namespace:*anything*' for 'read' inputs and the exact entered text and possible siblings/parents for 'write' inputs, during loading. so, typing 'character:' just to get the special 'character:*anything*' predicate is now simple and does not need a whole load wait to enter!
  • cleaned up some tag listbox code to handle parent selection and navigation better along with the new label type
  • greatly improved autocomplete search logic in the critical text search portion, collapsing it into one cleverer and more easily cancellable query rather than two or three simpler ones with potentially gigantic lists thrown back and forth
  • improved speed of autocomplete cancel for certain large lists with many siblings

  • file maintenance

  • the new file maintenance ui now shows scheduled jobs in a listctrl, and only shows jobs that have outstanding work. you can clear/do work on multiple selected jobs
  • the file manager should now try to guarantee at least 644 permission on file imports (previously, it was only trying to add 600, which lead to problems with nocopy ipfs running on another user etc...)
  • added a file maintenance job to check and fix file permissions
  • added a file maintenance job to regenerate similar files metadata
  • added a file maintenance job to check if a file should be in the similar files system--if it should and isn't, it is queued to get its metadata data regenerated, and if it is and shouldn't be, it is removed
  • the previous bulky similar files metadata regen job from the duplicates page is now removed, and any outstanding scheduled regen will be transferred to the new file maintenance manager on update

  • client api

  • added POST /manage_pages/focus_page, which makes the given page the current page in the main gui
  • added help and unit tests for this new call
  • client api is now version 9

  • the rest

  • fixed an issue recording media viewtimes when no max viewtime is set
  • fixed the new missingdirectory errors not printing the missing path
  • fixed an issue with some human-started repository actions waiting silently on bandwidth when it was not intended (e.g. account refresh)
  • export folders now raise proper errors and pause themselves if their path is not set, does not exist on the file system, or is not a directory (previously, they silently stopped work without error)
  • cleaned up some misc import folder code, and put in additional protections to the delete/move code to ensure folders cannot be so actioned if they somehow end up in the path import queue
  • when unpinning a file or directory from ipfs, the clientside service now first checks that the current daemon considers it pinned (previously, this 500 errored when the object was not pinned due to a reinitialised daemon etc...)
  • fixed an issue with the new ipfs path translation control, which was forgetting values when the clientside path was outside of the default db structure
  • media objects that transition from trashed to physically deleted but remain in view will now correctly be aware of their complete previously-deleted status (rather than being simply remote, as they were before until a client restart)
  • improved some of the recent duplicates db update code to pre-optimise the new tables on update (some users were getting slow behaviour due to mis-scheduled analysis maintenance)
  • extended the new panel system to deal with custom button panels and moved the duplicate filter 'commit and continue?' dialog to the new panel system
  • moved the archive/delete and duplicate filter 'commit and finish' dialog to the new panel system
  • wrote a new question panel for the typical yes/no dialog used across the program and started a cleanup job to migrate all 140-odd instances of this over
  • fixed an issue where a program instance that quit due to a user deciding to leave an already running instance in place would clear the original instance's 'running' file in its shutdown, meaning subsequent runs would charge ahead and hit 'database is locked' problems on db init!
  • wrote a new 'similar files metadata generation report mode' to provide debug info on this cpu/gpu intensive routine
  • added 'why use sqlite?' entry to the help faq, with a link to prkc's excellent document about the subject, https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md
  • also added prkc's excellent Linux package requirements information to the 'running from source' help page
  • fixed some old py 2.7 references in running from source help and an old link in ipfs help
  • moved the 'file viewing statistics' menu down on the database menu
  • fixed some dialog Escape key event handling
  • fixed some ui ancestory testing code
  • improved some misc similar files system code

• version 359

  • ipfs nocopy

  • wrote a new panel to better show ipfs daemon status and added it to the review and manage ipfs service panels
  • added nocopy config review and enable status and buttons to this new panel
  • added an EXPERIMENTAL 'use nocopy' checkbox to the ipfs manage services panel
  • added accompanying WEWLAD path translation ui to enable nocopy when your hydrus media storage paths are inaccessible to the ipfs daemon for nocopy purposes. a help button explains this more--it currently needs some symlinking, so non-advanced users should stay away
  • if everything is set up, ipfs nocopy seems to work! I am not totally happy about the setup required here, so feedback from advanced ipfs-fluent users would be appreciated and we can iterate on this
  • improved stability of ipfs daemon/version checking code

  • file maintenance

  • wrote some proper file maintenance ui under database->maintain->review scheduled file maintenance!
  • for existing work, the new file maintenance ui shows how much work is scheduled for each job type and lets you cancel that work or run it manually
  • for new work, the new file maintenance ui lets you queue up work of any type for files you select with the standard tag autocomplete search interface! you can schedule all pngs to be rescanned in case they are truly apngs, or regen thumbs for all files imported before a certain date, or whatever you wish. you can also queue up repository update files
  • the file maintenance manager can now deal with repository update files when it does a complete file metadata regen
  • the file maintenance manager now takes responsibility for checking file presence and file integrity. the old 'check file integrity' options under database->maintenance, which did all files in one go, are now gone
  • file integrity checks will now always export broken files and missing/broken files' known urls to .txt files to your db_dir/missing_and_invalid_files. appropriate popups and log data will be sent as well. also, the known urls will be both exported on a per-file .txt basis and appended to one unified .txt
  • if a file now fails to parse on a metadata reparse, it is now automatically checked for file data integrity
  • if a repository encounters a missing, invalid, or incorrect filetype update during update processing, it now schedules all updates in the repo to be appropriately rescanned by the file maintenance manager
  • if the storage subdirectory directory does not exist on a client file path request or thumbnail-add attempt, a special error will now be raised with instructions to reconnect the location or shut the client down immediately
  • cleaned up some ffmpeg mime-detection logspam

  • duplicates

  • added several single-file thumbnail right-click dissolve/reset duplicate actions

  • - reset search status
  • - remove from duplicate group (if in one and not the king)
  • - dissolve duplicate group (if in a group)
  • - remove from alternate group (if in one)
  • - dissolve alternate group (if in one)
  • - clear false-positive relations (if it has some)
  • added some new code to deal with dissolution and member extraction at the db level
  • when a member is extracted from alternate group, its constituent files are now requeued for potential search
  • multi-selection duplicate right-click actions are now available to non-advanced-mode users
  • wrote some unit tests for the new dissolve/reset actions
  • cleaned up some misc duplicates code

  • the rest

  • fixed a recent bug in the file lookup script GET call--I apologise for the mistake
  • the main gui page tab menu now lets you sort page tabs by the number of files they have
  • deviant art seem to be rolling out a new page format. this week hydrus introduces completely new deviant art downloader objects that, fingers crossed, will update any existing users smoothly and also provide new tag search functionality. users who are still logged in may still be getting the old page format. if this is you, and this update does not work (although I _think_ it should, even so), please try clearing your existing login and logging in again
  • new deviant art login script, artist + tag GUGs, gallery url classes, file and gallery parsers
  • updated the shimmie file page parser to pull source time and md5
  • improved the 'process now' advanced button to only focus on actual specific outstanding processing. previously, it was also checking for new metadata when due, which, when the server was not available, could seemingly idle for a time before actually processing updates due to the new delaying connection retry code
  • wrote a new 'file import report mode' mode to help->debug->report modes
  • fixed a progress display issue with the janitorial petitions processing page
  • improved accuracy of sibling and parent petition counts, and properly capped them at 1000
  • mapping petitions are now grouped by namespace, and will come in more manageable chunks
  • fixed the server launch-and-init test debug code
  • misc string-to-string control improvements to support the new ipfs edit ui
  • removed the old 'continual tag archive sync' legacy code from tag services, which has been semi/non-functional for a long time
  • cleaned up the annoying separator hanging on the end of certain tag right-click menus
  • cleared out the 'Exception ignored in' spam that is often printed after the log closes

• version 358

  • duplicates

  • the final large data storage overhaul work of the duplicates work big job is done--potential duplicate information is now stored more sensibly and efficiently. potential pair information is now stored between duplicate file groups, rather than files themselves. when duplicate file groups are merged, or alternate or false positive relationships set, potentials are merged and culled appropriately
  • your existing potential data will be updated. the current potential pairs queue size will shrink as duplicate potential relationships are merged
  • the duplicate filter now presents file kings as comparison files when possible, increasing pair difference and decision value
  • potential pair information is now stored with the 'distance' between the two files as found by the similar-files search system. the duplicate filter will serve files with closer distance first, which increases decision value by front-loading likely duplicates instead of alts. distance values for existing potential pair info is estimated on update, so if you have done search distance 2 or greater and would like to fill in this data accurately to get closer potentials first, you might like to reset your potential duplicates under the cog icon (bear in mind this reset will schedule a decent whack of CPU for your idle maintenance time)
  • setting alternate relationship on a pair is now fixed more concretely, ensuring that in various search expansions or resets that the same pair will not come up again. this solves some related problems users have had trying to 'fix' larger alternate groups in place--you may see your alternates compared one last time, but that should be the final go. these fixed relationships are merged as intra-alternate group members merge due to duplicate-setting events
  • a variety of potential duplicates code has been streamlined based on the new duplicate group relationship
  • improved how a second-best king representative of a group is selected in various file relationship fetching jobs when the true king is not permitted by search domain
  • one critical part of the new potential duplicates system is more complicated. if you experience much slower searches or count retrievals IRL, please let me know your details
  • expanded duplicates unit tests to test potential counts for all tested situations
  • fixed a bug where alternate group merging would not cull now-invalid false-positive potential pairs

  • the rest

  • updated the default pixiv parser to work with their new format--thank you to a user for providing this fix
  • fixed the issue where mouse scroll events were not being processed by the main viewer canvas when it did not have focus
  • file page parsers that produce multiple urls through subsidiary page parsers now correctly pass down associated urls and tags to their child file import items
  • updated to wx 4.0.6 on all built platforms--looks like a bunch of bug fixes, so fingers-crossed this improves some stability and jank
  • updated the recent server access-key-arg-parsing routine to check access from the header before parsing args, which fixes an issue with testing decompression bomb permission on file POST requests on the file repository. generally improved code here to deal more gracefully with failures
  • the repositories now max out at 1000 count when fetching pending petition counts (speeding up access when there are large queues)
  • the repositories now fetch petitions much faster when there are large queues
  • frames and dialogs will be slightly more aggressive about ensuring their parents now get focus back when they are closed (rather than the top level main gui, which sometimes happens due to window manager weirdness)
  • rewrote a bad old legacy method of refocusing the manage tags panel that kicks in when the 'open manage tags' action is processed by the media viewer canvas but the panel is already open
  • hitting 'refresh account' on a paused service now gives a better immediate message rather than failing after delay on a confusing 'bad login' error
  • improved login errors' text to specify the exact problem raised by the login manager
  • fixed a problem in the duplicates page when a status update is called before the initial db status fetch is complete
  • the manage tag siblings panel now detects if the pair you wish to add connects to a loop already in the database (which is a rare but possible case). previously it would hang indefinitely! it now cancels the add, communicates the tags in the loop, and recommends you break it manually
  • added a link to https://github.com/cravxx/hydrus.js , a node.js module that plugs into the client api, to the help
  • a variety of user-started network jobs such as refreshing account and testing a server connection under manage services now only attempt connection once (to fail faster as the user waits)
  • the 'test address' job under manage services is now asynchronous and will not hang the ui while it waits for a response
  • fixed some unstable thread-to-wx code under the 'test access key' job under manage services
  • improved some file handling to ensure open files are closed more promptly in certain circumstances
  • fixed some unstable thread-to-wx communication in the ipfs review services panel
  • improved the accuracy of the network engine's 'incomplete download' test and bandwidth reporting to work with exact byte counts when available, regardless of content encoding. downloads that provide too few bytes in ways that were previously not caught will be reattempted according to the normal connection reattempt rules. these network fixes may solve some broken jpegs and json some users have seen from unreliable servers
  • fixed watcher entries in the watcher page list not reporting their file and check download status as they work (as the gallery downloader does)
  • the client api will now deliver cleaner 400 errors when a given url argument is empty or otherwise fails to normalise (previously it was giving 500s)
  • misc cleanup

• version 357

  • client api

  • the client api can now receive the access key through a GET or POST parameter rather than the header
  • the client api now supports GET /session_key, which provides a temporary key that gives the same access as its permanent access key with the Hydrus-Client-API-Session-Key name through header or GET/POST param. it expires after 24 hours of inactivity or if the client is restarted
  • the GET /manage_pages/get_pages call now returns the unique 'page_key' identifier that will be useful in future page management when multiple pages share a name
  • the POST /add_urls/add_url command now takes 'destination_page_key' to exactly specify which page you would like a URL to end up on. if the page is not found, or it is the incorrect type, the standard page selection/creation rules will apply
  • cleaned up some serverside request processing code
  • cleaned up some misc client api permission checking code
  • updated client unit tests to check the new changes
  • updated client api help to reflect the new changes
  • cleaned up some GET and POST parameter parsing
  • client api version is now 8

  • shortcut and hover window fixes

  • moved the canvas shortcut processing code more towards the new shortcut system
  • the OS X shortcut-in-media-viewer issue, which was being boshed in a similar way to the main gui last week, should now be fixed
  • when the hover windows have focus, they now pass shortcuts up to the canvas parent more reliably
  • removed a legacy menu highlight-tracking system that was malfunctioning and generally throwing a slow-memory-leaking wrench in several places, particularly some non-Windows situations
  • the 'menubar is open' test code is now only active for Windows. the other platforms have mixed reliability with menubar open/close events
  • some related OS X hover-window flickering and hiding-under the main page problems (having problems due to thinking menus were open) are also fixed
  • some hover window flicker on certain focus changes due to clicking focus windows should be fixed
  • hover windows now try to size themselves a little better on init, which reduces some initial flicker or false-positive single-frame display on some systems
  • extended the hover report mode to report some 'ideal' pos/size info as well
  • under file->shortcuts, custom shortcuts are now hidden for non-advanced-mode users

  • the rest

  • the popup message toaster now always shows its 'dismiss all' summary bar whenever any messages are being displayed. the summary bar now also has a ▼/▲ button to collapse/expand its messages!
  • added duplicate comparison score options (under options->duplicates) for the new jpeg quality estimator
  • fixed the default duplicate comparison score values, which appeared to be reversed for higher vs much higher--they will be reset to these new defaults on update, so recheck them if you prefer different
  • in manage tag siblings and parents, the filename tagging dialog, and some misc options panels, tag autocomplete input controls now have 'paste' buttons to make entering many results much easier
  • to reduce update flicker, the downloader and watcher pages do not list seconds in their 'added' column ('12 minutes 24 seconds ago' is now '12 minutes ago')
  • improved clipboard access cleanup on in-clipboard errors, which was sometimes leading to error popups or clipboard lockup
  • rather than the simple 'yes', the review bandwidth usage dialog now puts the waiting estimate (like '12 minutes 50 seconds') in the 'blocked?' column
  • improved external program launch code for non-Windows to remove hydrus-specific LD_LIBRARY_PATH completely when no OS default exists to restore it to. this should fix ffmpeg connection for certain installs
  • fixed a rare bug when initial media results of a page failed to load due to a subset of unexpectedly unfetchable file records
  • gave the rare 'ui freezup on dialog close' event yet another pass. closing via escape key should now be immune to this
  • 'remote' files that were once in the client but since deleted now have the 'trashed' icon and will state so on their right-click info summary lines
  • fixed various instances where selection-from-list dialogs were failing to sort their list based on underlying data object uncomparibility. an example of this was when selecting which queries to pull from a separating subscription
  • on the edit parser panels, fetching test data successfully via the quick button or the manual URL entry will now set that URL in the example parsing context
  • on the edit parser panels, the subsidiary page parser's separation formula now launches with the correct example data (the original data from the parent dialog, rather than the post-separated data) on which to test separation. this should nest correctly for multiple subsidiary page parsers
  • to reduce server load spikes, clientside petition processing now approves very large mapping and file petitions (such as a petition to delete one tag from 50k files) as a sequence of smaller chunks

• version 356

  • duplicates

  • moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file
  • as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A)
  • the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this
  • the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best
  • as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed
  • added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file
  • the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon
  • due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate!
  • setting duplicate relationships will overrule false positive or alternate relationships already in place
  • manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident
  • all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together
  • fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes
  • refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability
  • potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons

  • the rest

  • fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups
  • the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information
  • fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here
  • the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs
  • when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button!
  • cleaned up some file maintenance manager maintenance locking and shutdown cancel logic
  • moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate
  • misc shutdown logic cleanup
  • added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due
  • the 'regenerate' thumbnail menu is also available to all users
  • jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels
  • the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays
  • the duplicate filter now handles bad jpeg quality estimations gracefully
  • cleaned up some ffmpeg communication code
  • the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get
  • the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error
  • dialogs and windows on the new panel system now support a new pre-close tidying system
  • the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close
  • regular gui pages now support a new pre-close tidying system
  • search pages will now cancel any pending search results loading or tag autocomplete queries on close
  • improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation
  • the network engine now waits significantly longer--60s--on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster
  • I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers
  • improved hover window report mode to state more focus info in case the above is insufficient
  • to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30
  • updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution
  • sped up tag parents initialisation
  • repositories now group tag sibling and parent petitions by the parent/better tag's namespace
  • removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing
  • the log now flushes itself to disk every 60s rather than 300s
  • misc fixes and cleanup

• version 355

  • duplicates

  • retuned the new alternate and false positive setting code to be less aggressive about removing potential pairs. users with alternate groups who updated to 354 may have lost some intra-alternate potential pairs, which I expect to fill back in once the potential pairs system is updated
  • the duplicate filter center-right hover window now has a trash button, which is moved to its own row with the cog icon where they will fit on thinner windows better
  • users who are in advanced mode now have access to duplicate merge options for 'alternates' again--but be careful with this, it is advanced. if you had merge options for alternates set up before, your old options _should_ return, but make sure to check it
  • if you are in advanced mode, the duplicate filter can now show an experimental jpeg quality comparison statement. if this works out, I will add custom scores and improve this otherwise based on feedback , so comments would be appreciated

  • client api

  • the client api has a new 'manage pages' permission!
  • /manage_pages/get_pages now fetches a structure of the current page session! there's some help for it
  • in future, I expect to add a 'select page' command and get/remove/highlight URL(s) for downloader pages, and reveal the unique page identifier for better targeting here and for add_url commands
  • client api is now version 7

  • the rest

  • fixed a stupid issue related to drawing collection thumbnails that was causing a lot of errors. I apologise for the inconvenience and have added pre-build tests to catch these simple mistakes in future
  • added a clipboard url watcher! there are two options now under network->downloaders--one for watcher urls, and one for all other _recognised_ urls (i.e. post and gallery urls, and file urls that have matching url classes). it checks every second, sends urls to pages according to the same rules as a drag and drop event, and will handle newline-separated lists of urls
  • when the client is minimised and needs to create a new downloader page because of an 'add url' command through this new clipboard watcher or the client api, it will now queue up the url and its page creation for when the client is next restored (page creation while minimized causes layout bugs). it returns an appropriate result text to the client api in this case
  • added four 'show_and_focus_manage_tags_XXX' shortcuts to the 'main_gui' shortcut set that let you select and focus the four possible suggested tags panels on the manage tags dialog. these are slightly special focus events that do some 'first tag selection' work as well and should let you make the whole process keyboard-only
  • added a 'focus_media_viewer' shortcut to the 'main_gui' shortcut set that focuses the media viewer from a media viewer's manage tags window
  • file viewing statistics gets an update: it now has its own options page where you can set min/max for preview and media viewing times. the global turn on/off control is moved from database menu to this page, as is a new control for enabling it on the duplicate filter (where you will be flicking back and forth and perhaps do not want lots of views recorded). duplicate filter file stats now default to disabled
  • added a new command to database->file viewing statistics that will cull your current stats based on the new min/max values to compensate for previous accidental '3day 4hour' preview view that snuck in. a yes/no dialog explains how it works before it goes
  • wrote a 'mixin' class to let my buttons automatically note current shortcut mapping information to the bottom of button tooltips
  • the duplicate commands and a variety of other media viewer buttons that work in the new shortcut system now report their current shortcuts in their tooltips!
  • quick-entering 'namespace:' in a search page's tag autocomplete input will now automatically swap in the special 'namespace:*anything*' predicate
  • cleaned up code around setting a custom temp_dir through the "temp_dir" command, and made it test the given path is indeed a directory and writable-to. if not, the program dumps out with an error popup
  • the client and server now similarly test that the directory db is a dir and writable-to
  • rearranged some critical boot error handling code and made the client's boot error handling throw up some ui
  • fixed the client's repairdb function when needing to regen autocomplete caches
  • autocomplete regen now publishes status updates to the splash screen
  • fixed the client's repairdb to wait correctly for the ui in case the user wants to bail before repair
  • the client's repair db now repairs the new local_tags_cache table if missing
  • fixed an issue with showing the manage logins panel when the domain had some unusual cookie expiry numbers
  • to reduce initial load gui-event crunch (which affects some systems' layout reliability), initial session load now happens after a 0.5s delay
  • cleaned up some image load error handling
  • when FFMPEG version information cannot be understood in help->about, a popup note appears and debug information is printed to the log
  • the advanced content update panel now puts up a 'working...' dialog while it processed a job
  • in the client
  • all 4XX and 5XX network exceptions' error texts are now prepended with the actual status code
  • extended the hover window debug report mode
  • improved some misc temp and permission testing code
  • improved some misc testing code

• version 354

  • duplicates important

  • duplicates 'false positive' and 'alternates' pairs are now stored in a new more efficient structure that is better suited for larger groups of files
  • alternate relationships are now implicitly transitive--if A is alternate B and A is alternate C, B is now alternate C
  • false positive relationships remain correctly non-transitive, but they are now implicitly shared amongst alternates--if A is alternate B and A is false positive with C, B is now false positive with C. and further, if C alt D, then A and B are implicitly fp D as well!
  • your existing false positive and alternates relationships will be migrated on update. alternates will apply first, so in the case of conflicts due to previous non-excellent filtering workflow, formerly invalid false positives (i.e. false positives between now-transitive alternates) will be discarded. invalid potentials will also be cleared out
  • attempting to set a 'false positives' or 'alternates' relationship to files that already have a conflicting relation (e.g. setting false positive to two files that already have alternates) now does nothing. in future, this will have graceful failure reporting
  • the false positive and alternate transitivity clears out potential dupes at a faster rate than previously, speeding up duplicate filter workflow and reducing redundancy on the human end
  • unfortunately, as potential and better/worse/same pairs have yet to be updated, the system may report that a file has the same alternate as same quality partner. this will be automatically corrected in the coming weeks
  • when selecting 'view this file's duplicates' from thumbnail right-click, the focus file will now be the first file displayed in the next page

  • duplicates boring details

  • setting 'false positive' and 'alternates' status now accounts for the new data storage, and a variety of follow-on assumptions and transitive properties (such as implying other false positive relationships or clearing out potential dupes between two groups of merging alternates) are now dealt with more rigorously (and moreso when I move the true 'duplicate' file relationships over)
  • fetching file duplicate status counts, file duplicate status hashes, and searching for system:num_dupes now accounts for the new data storage r.e. false positives and alternates
  • new potential dupes are culled when they conflict with the new transitive alternate and false positive relationships
  • removed the code that fudges explicit transitive 'false positive' and 'alternate' relationships based on existing same/better/worse pairs when setting new dupe pairs. this temporary gap will be filled back in in the coming weeks (clearing out way more potentials too)

  • several specific advanced duplicate actions are now cleared out to make way for future streamlining of the filter workflow

  • removed the 'duplicate_media_set_false_positive' shortcut, which is an action only appropriate when viewing confirmed potentials through the duplicate filter (or after the ' show random pairs' button)
  • removed the 'duplicate_media_remove_relationships' shortcut and menu action ('remove x pairs ... from the dupes system'), which will return as multiple more precise and reliable 'dissolve' actions in the coming weeks
  • removed the 'duplicate_media_reset_to_potential' shortcut and menu action ('send the x pairs ... to be compared in the duplicates filter') as it was always buggy and lead to bloating of the filter queue. it is likely to return as part of the 'dissolve'-style reset commands as above
  • fixed an issue where hitting 'duplicate_media_set_focused_better' shortcut with no focused thumb would throw an error
  • started proper unit tests for the duplicates system and filled in the phash search, basic current better/worse, and false positive and alternate components
  • various incidences of duplicate 'action options' and similar phrasing are now unified to 'metadata merge options'
  • cleaned up 'unknown/potential' phrasing in duplicate pair code and some related duplicate filter code
  • cleaned up wording and layout of the thumbnail duplicates menu

  • the rest

  • tag blacklists in downloaders' tag import options now apply to the parsed tags both before and after a tag sibling collapse. it uses the combined tag sibling rules, so feedback on how well this works irl would be appreciated
  • I believe I fixed the annoying issue where a handful of thumbnails would sometimes inexplicitly not fade in after during thumbgrid scrolling (and typically on first thumb load--this problem was aggravated by scroll/thumb-render speed ratio)
  • when to-be-regenerated thumbnails are taken off the thumbnail waterfall queue due to fast scrolling or page switching, they are now queued up in the new file maintenance system for idle-time work!
  • the main gui menus will now no longer try to update while they are open! uploading pending tags while lots of new tags are coming in is now much more reliable. let me know if you discover a way to get stuck in this frozen state!
  • cleaned up some main gui menu regeneration code, reducing the total number of stub objects created and deleted, particularly when the 'pending' menu refreshes its label frequently while uploading many pending tags. should be a bit more stable for some linux flavours
  • the 'fix siblings and parents' button on manage tags is now a menu button with two options--for fixing according to the 'all services combined' siblings and parents or just for the current panel's service. this overrides the 'apply sibs/parents across all services' options. this will be revisited in future when more complicated sibling application rules are added
  • the 'hide and anchor mouse' check under 'options->media' is no longer windows-only, if you want to test it, and the previous touchscreen-detecting override (which unhid and unanchored on vigorous movement) is now optional, defaulting to off
  • greatly reduced typical and max repository pre-processing disk cache time and reworked stop calculations to ensure some work always gets done
  • fixed an issue with 'show some random dupes' thumbnails not hiding on manual trashing, if that option is set. 'show some random dupes' thumbnail panels will now inherit their file service from the current duplicate search domain
  • repository processing will now never run for more than an hour at once. this mitigates some edge-case disastrous ui-hanging outcomes and generally gives a chance for hydrus-level jobs like subscriptions and even other programs like defraggers to run even when there is a gigantic backlog of processing to do
  • added yet another CORS header to improve Client API CORS compatibility, and fixed an overauthentication problem
  • setting a blank string on the new local booru external port override option will now forego the host:port colon in the resultant external url. a tooltip on the control repeats this
  • reworded and coloured the pause/play sync button in review services repository panel to be more clear about current paused status
  • fixed a problem when closing the gui when the popup message manager is already closed by clever OS-specific means
  • misc code cleanup
  • updated sqlite on windows to 3.28.0
  • updated upnpc exe on windows to 2.1

• version 353

  • duplicate filter

  • duplicate action options no longer handle file deletion
  • renamed 'not duplicates' across the program to 'not related' or 'false positive'
  • 'alternates' and 'not related/false positive' duplicate actions no longer have duplicate action options. no merge content update now occurs on these actions
  • the duplicate filter hover panel now splits 'this is better' decisions into two buttons--whether to delete or keep the worse file
  • when selecting 'custom action' in the duplicate filter hover panel, it now asks if you would like to delete the current file, the other file, or both
  • the 'duplicate_filter_this_is_better' shortcut action will be auto-updated to 'duplicate_filter_this_is_better_and_delete_other'. an alternate 'duplicate_filter_this_is_better_but_keep_both' is now also available
  • the 'duplicate_filter_not_dupes' shortcut action will be auto-updated to 'duplicate_filter_false_positive'
  • separated the buttons on the duplicate filter hover panel to more carefully split 'yes, files are duplicates' vs other decisions
  • in prep for the duplicate db overhaul, refactored all PHash search code and Duplicate management code apart
  • misc other prep work for duplicate db overhaul

  • file maintenance

  • wrote a new unified manager to handle various long-term file maintenance tasks like regenerating file metadata and thumbnails
  • options to govern how this manager can run are now in options->maintenance and processing. you can enable it for idle and shutdown maintenance time and give it a throttle to limit how fast it will work on files, defaulting to 200 per day
  • unified the previous db-level attempts at file maintenance to the new system, which supports async job queueing, and moving regen code up to the new manager, out of the db lock
  • unified a variety of file and thumbnail regen code to work through the new simpler and saner path
  • the right-click->regen thumbnail commands now run through the new manager and no longer need a modal popup. you can keep browsing while they work. they will also not hang the ui as the old system could on big jobs
  • when right-click->regenning on more than 50 thumbnails, you now get a dialog asking if you want to do the job now or put it off later
  • file maintenance tasks can now run in shutdown time! you will get previews of the jobs with file counts and status progress reports on the shutdown splash
  • cleaned up some file extension renaming and dupe-removing code
  • in future, I will move the current file integrity check to this new system and have some ui to prompt and set up other big jobs, like fixing various historical misparsing issues
  • thumbnail resizing during thumbnail fade that resizes down is now more efficient
  • moved the ClientFilesManager to ClientFiles.py

  • the rest

  • the 'manage upnp' dialog now moves the duplicated external ip display from the column up to the status text at the top. it fetches the ip after the initial mappings fetch is done. this ip is no longer affected by the external host override option
  • cleaned up options->connection page and removed the now defunct external host override option
  • the manage services page for the local booru now has optional override for scheme, host, and port for the 'copy external url' function
  • fixed an issue with the recent 'collect by' session saving where a restored session that needed a collect was not sorted
  • fixed an issue with collections being sorted by approx bitrate
  • added a new checkbox to options->sort/collect to set it so the default sort updates every time you choose a new sort anywhere
  • fixed an issue with 'remove trashed files from view', which was incorrectly removing on 'all local files' pages
  • the 'all local files' file domain, which is frequently confusing to new users, is now no longer an option for new file pages or the autocomplete file domain if the user is not in advanced mode
  • the client now searches for versions of urls both with and without a final '/' character when looking up file url import status at the db level and in import lists. system:known_url is unfortunately still an inefficient mess
  • improved how the server code deals with some connectionLost errors
  • cleaned up and unified some older dialog button code
  • fixed a problem in manage tag siblings when petitioning existing pairs and then cancelling when asked for a reason
  • fixed a miscount issue when uploading pending tags while many new tags are coming in. progress would sometimes be -754/1,234, ha ha
  • db maintenance, repository sync, and file maintenance processing will all now wake on a force idle mode call
  • deleted some old code
  • misc fixes and cleanup
  • some misc gui layout fixes

• version 352

  • the client now supports importing .ico files! (.cur should be supported too)
  • finally, 'collect by' is saved for sessions! if your default collect by previously included ratings services, it will forget them this one time--please reset it under the options->sort/collect
  • fixed the issue where the media viewer's hover windows were hovering over child dialogs (manage tags, ratings, or known urls)
  • improved some os x hover window focus handling for the new always-on-top duplicate action window
  • the entries on the 'sort by' list on gui pages are now subcategorised better. it should be a bit easier to find what you are looking for
  • the 'sort by file: approximate bitrate' sort option now sorts still images as well by filesize / num_pixels
  • to reduce confusion, sort by mime and system:mime are now renamed to 'filetype'
  • fixed an issue where the 'unclose_page' shortcut was restoring pages in reverse order (unclosing least-recently-closed-first rather than most-recently-closed-first)
  • improved rigour of video framerate estimation
  • stopped the video metadata parser from opting to manually frame count videos with size >128MB or num_frames estimate >2,400
  • fixed the forced manual frame count to deal with frame counts >9999
  • the 'ffmpeg not found' error on file import will now put up a popup message once per boot informing you of this problem more broadly and steps to address it
  • fixed some underreporting issues with subprocess_report_mode
  • fixed an issue with some yes/no dialogs returning 'no' on escape/window_close_button rather than 'cancel', which affected cancelability some db maintenance questions
  • fixed an issue where media that fitted the media viewer canvas width or height exactly at 100% zoom would not respond to zoom switch events to restore non-100% zoom to 100%
  • when a local server's CORS mode is turned on, Access-Control-Allow-Origin is now correctly added to GET/POST requests with an Origin request header
  • improved reliability of some timestamp rendering code, which should help some users who had trouble opening cookies management page after malformed cookie import
  • I believe I fixed an issue with 'open externally' on certain custom paths where the external program could spawn without an ui (flash projector did this). please let me know if your 'open externally' calls start making terminal windows everywhere
  • fixed a runtime stability issue with the new duplicates page and slow-updating counts that come in after the page has been deleted

• version 351

  • wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements
  • the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance
  • improved some hover window focus display calculations to play with the new always-on-top tech
  • both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result
  • core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly
  • the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it
  • unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug
  • gave all image processing a refactor and general cleanup pass, deleted a bunch of old code
  • wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate
  • sped up how fast the tag parent structure builds itself
  • the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner
  • similar-files metadata generation now discards blank frames more reliably
  • subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in
  • made the disk cache more conservative on non-pre-processing calls
  • cleaned up some file import code, moving responsibility from the file locations manager to the file import object
  • updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable
  • I believe I fixed a rare vector for the 'tryendmodal' dialog bug
  • fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name--duplicate-named objects in this case will now be discarded
  • unified url_match/url_class code differences to url class everywhere
  • updated some common db list selection code to use new python string formatting
  • plenty of misc code cleanup

• version 350

  • the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long
  • fixed the duplicate pairs filter's 'ghost pair' issue. it was failing, when 'both files' was unchecked, to remove pairs that included one file that was non-local. this accidental inclusion resulted in incorrect non-zero count and filter/random pairs that could not display correctly
  • insulated against potential future iterations of this problem (likely that one of the pair was deleted by another process while a filter is ongoing), with a notification and graceful exiting of the duplicate filter while saving progress
  • the 'show random duplicates' button now puts the 'base' of the group (to which all the others are potentially matched) as the first thumbnail
  • added a new 'advanced file deletion' section to 'files and trash' options page to turn on a new advanced dialog and set custom file deletion reasons
  • if this new dialog is turned on, a delete event from thumbnail grid, regular media viewer, or the duplicate filter's manual delete will launch it. it permits you to delete physically (skipping trash) in one step or delete physically without leaving a deletion record (for easier later re-import) and choose one of the deletion reasons in the new options panel or set a one-time custom reason
  • export folders now have more run-controls: 'run regularly', 'paused', and 'run now'
  • the file menu now has a 'run export folder now' submenu just like for import folders-- it is simple now to set up an export folder that only runs when you tell it to
  • updated the on-boot missing file folder recovery process to automatically 'heal' file location mappings when a missing folder is actually in a location that is known (essentially, you can now manually move a bunch of folders from one known location to another while the client is off and it will recover automatically now). error dialogs will appear in this case summarising the problem and proposed fixes with a chance to bail out if you do not want it to happen
  • added a new frame type to 'gui' options page called 'regular_center_dialog' for yes/no style dialogs that are better in the center of the parent window
  • the custom web browser launch path and file type 'open externally' paths are moved from 'files and trash' to a new 'external programs' options page
  • as the superior '--temp_path' program launch parameter now exists for both client and server, I have removed the limited 'BUGFIX: temp folder override' option from the client's 'files and trash' page and use in the actual code. if this option was important to you, please migrate to the --temp_path launch parameter, which covers temp usage more comprehensively
  • as the artstation downloader is now non-functional, apparently by a cloudflare issue, the default gug for new users (and anyone with artstation set atm) is now safebooru
  • added dolphin file manager add-on link to the client api help
  • some misc file metadata fetching cleanup

• version 349

  • duplicate filter

  • the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!

  • the duplicate filter page has multiple layout changes as a result

  • the main management area is now split into two pages--'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found
  • the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'
  • the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis
  • the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating...' text while they work). both now have explicit refresh buttons to force them to update
  • the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number
  • the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes
  • the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)
  • misc duplicate file db code cleanup and refactoring
  • while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context
  • making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair

  • manage tags

  • the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires
  • the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions
  • the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists

  • the rest

  • gave video metadata parsing another pass--it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames
  • manual video frame count, when needed, is now faster and produces better error text
  • fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information
  • significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos
  • fixed an issue where regenerated file metadata was not propagating up to the ui level in real time
  • cleaned up some thumbnail cache initialisation code
  • the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page
  • to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed
  • all three fields of text on serialised pngs now wrap, and they pad a little better as well
  • added a new option to the 'gui pages' options page to force input text box focus on page changes
  • fixed a small type issue with the server's session cookie code and some new library versions

• version 348

  • wrote some OR search help to 'getting started with tags' help page
  • wrote a new multi-reader, single-writer lock object for the client file manager, along with some unit tests for it.
  • updated the file and thumbnail access and regen and maintenance code to use the new lock. various access is now faster when available and overall safer. there is still work to do here
  • adjusted file import to be less aggressive about locking, which should reduce some file/thumbnail access lag during heavy imports
  • the thumbnail space estimate in the migrate database dialog is now more adaptive to the new more flexible thumbnail size system and specificies better that it is an estimate
  • the client api's /add_tags/add_tags call now collapses siblings and expands parents on an add/pend call. this can be turned off by setting the new optional parameter 'add_siblings_and_parents' to false. the help is updated regarding this and the client api version is incremented to 6
  • fixed the client api's /add_tags/add_tags call for the 'hashes' parameter, which was failing to parse, and added an accidentally missing unit test to check this in future
  • the client local services (the booru and client api) now have two new options under their 'manage services' panel: 'support CORS', which turns on cross-orogin support (which is experimental for now, so defaults to False), and 'logs requests', which controls whether your log will be spammed with request reports (this also defaults to False), which should clear up some 100MB+ log hassle for people using the Hydrus Companion browser add-on
  • hydrus services now respond correctly (albeit sparsely) to OPTIONS requests, and if CORS is enabled, to CORS OPTIONS requests. there are unit tests for this that seem to work ok, but I think we'll need to verify it irl
  • finished a first version bitmap manager to handle all wx bitmap creation and destruction, including recycling mid-steps
  • updated all simple wx bitmap creation and destruction calls across the client to use the new bitmap manager, improving stability and saving some CPU
  • fixed some incorrect button alignment flags that were causing problems for clients set to assert check these values
  • added a new yiff.party file url class to the defaults that matches a new file attachment format
  • updated the 'url' content parser so if a parsed url is in the form 'summary_text url', as some booru source fields sometimes specify, the preceding summary text is removed, cleaning up the resultant url
  • silenced an old server connection lost error that was needlessly loud
  • silenced the client network engine from additionally log-printing SizeException errors when a downloading file (usually gif) exceeds file import options rules
  • improved misc window destruction code
  • updated supported mime list in 'getting started with files' help and website index
  • misc cleanup

• version 347

  • or search

  • under construction OR predicates now present at the top of the regular tag results list, prepended with 'OR: ', and skipping default selection
  • this new OR line is enter-able, which will present it as-is, rather than adding new preds
  • hitting escape on a 'search' tag input box that is empty but has an under construction or predicate will cancel the or pred
  • hitting escape on a 'search' tag input box otherwise should more reliably kill its focus when the dropdown is a float window
  • improved OR search efficiency significantly with dynamic OR search triggering based on other search predicates. OR searches including negated '-tag' components should be massively faster when paired with non-OR tag or file search predicates
  • I believe I fixed a search issue that would sometimes return insufficient results when OR preds are mixed with certain other combinations of tags
  • improved reliability of some thumbnail refresh calls
  • cleaned up a bunch of OR handling ui code

  • the rest

  • after previous weeks' experiments, wrote new double-layer thumbnail loading system--now too-small thumbs will quckly scale up fuzzily straight to screen, and then in the coming seconds, the nice regenerated full-size thumb will be made and drawn in place as ready. it presents much faster and looks better, but there is some cleanup to do here that I will tackle next week
  • all local file trashing events now record a context-appropriate deletion statement such as "Deleted from Media Viewer." this value is recovered in 'deleted' import status 'notes'. You will mostly see 'Unknown deletion reason.', for files deleted before this new system, but it will populate with appropriate info over time
  • fixed a search optimisation that was not cross-referencing with file domain, meaning for instance that bare system:rating calls were returning since-deleted files
  • upnp management window now uses new listctrl
  • cleaned up some old custom page-naming code
  • added a 'data' debug call to clear out all cached thumbnails and force an instant ui thumb reload
  • fixed the trash bmp misalignment, ha ha
  • removed e-hentai login script from the defaults, since this testing script is not appropriate for new users
  • dejanked some media viewer video transitions by cleaning up animation bar rendering and smoothing out video buffer initialisation
  • cleaned out some surplus subprocess wait calls that were hanging some systems on various 'open externally' calls
  • fixed multiple syncing problems with 'synchronise' export folders that produce files with subdirectories. subdirectory structures should now be synced correctly and empty folders deleted
  • export folders that collapse multiple file results to the same duplicated name should, after the next run, do less overwriting to this same name
  • if an export folder or the regular export dialog makes a file destination path that is above the chosen directory (e.g. if the path starts with ../ or ..\), the export job will error out with an explanation
  • big manual file exports _should_ be politer to the ui and cause fewer hangs
  • doing page tab drag and drops may have less post-drop ui jank on linux, continued feedback would be appreciated
  • moved 'reason' handling for all content updates to its own area, which neatens many content update data handling issues
  • fixed petitioning a tag via a shortcut, which had bad reason handling
  • fixed an issue with committing pending ipfs items that was overchecking service permissions
  • fixed some remaining bad wx code in the unit tests
  • misc file status reporting cleanup

• version 346

  • or search

  • extended the search predicate object to handle more OR stuff
  • extended the tag list to handle list objects that have multiple colours
  • extended the new OR search predicate to report multiple text-snippet-and-colour pairs based on sub-predicates
  • extended tag search input to handle prototype OR predicate creation--hold shift when you enter the tag, and it'll start an OR chain. shift-enter continues the chain, enter alone completes it
  • fleshed out the predicate unit tests to cover more of this
  • wrote unit tests for OR search predicates. it seems good!
  • improved some search logic to apply system preds better in certain edge cases and spend less CPU on OR-search-only searches

  • thumbnails

  • thumbnails will now queue for load in a more intelligent order based on estimated difficulty to regenerate, which will tend to put more thumbs on screen faster
  • the decision to regenerate a thumbnail from source is now tempered by how different the current thumbnail is from what is desired--the more similar the two sizes, the more (randomly) likely the client will decide to just use the current (resized) this time. this smooths out change-lag while limiting the number of really fuzzy thumbs you get. feedback on how this works IRL would be appreciated--it uses some voodoo distribution polling to figure it out, which I can definitely tweak
  • improved visual quality of thumbnail scale-up optimisations
  • fixed an issue where a multipage thumbnail grid would incorrectly recalculate the new virtual height after a thumbnail size change event, leading to a bit of invalid extra scrollspace (with noclip rendering errors) at the bottom
  • the thumbnail right-click menu's reparse files entry is now extended to a new 'regenerate' submenu with three options: reparse file and regen thumbs (the old action), force regen thumbs, and regen thumbs if wrong size!
  • the new 'regen if thumbs wrong size' action sends how many thumbs needed resize up to the popup window, as well
  • moved some old thumbnail regen code responsibility out of the db and into the files manager
  • cleaned out some old redundant file/thumbnail code
  • cleaned and refactored a bunch of general image handling and resizing code

  • the rest

  • fixed some bad serialisation code that was making file search objects set their 'include current tags' value to false/true on interleaving loads. on this update, all 'include current tags' values are blanket reset to true
  • fixed an issue that was drawing animation canvases pure white on various media update events
  • extended manage urls dialog to support multiple files when launched from a selection of thumbnails. there is a warning in this case, noting that only gallery-style urls are appropriate to be added to multiple files
  • manage urls dialog now supports multiple selections, including shift-select, and accepts delete key presses for easy mass deletion
  • when you ask the database migration dialog to move some files, it now pops up a confirmation dialog that also asks if you would like to limit the max time for the job as 10, 30, or 60 minutes
  • improved file permission setting code across the program to be more sensible for non-Windows
  • if you are a non-Windows user and were hit with directory permission problems last week on the thumbnail update--which resulted in the rxx directories not being deleted--the update this week will attempt to do the delete again, this time correcting the now missing execute permission bit. if it finds outstanding rxx directories to delete, it will give a popup beforehand summarising the situation and giving you a chance to bail out
  • fixed yet another problem that was stopping client api url requests from finding the correct page by name
  • when a client api url request includes fixed tags, these tags should now propagate in all scenarios where the single url produces multiple files
  • updated sqlite dll and console for windows
  • misc fixes and cleanup

• version 345

  • or search

  • set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support
  • started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way--we'll work on this as we discover the most common inefficiencies

  • thumbnails

  • the client no longer uses both 'master' and 'resized' thumbnails--it uses a single, smarter thumbnail
  • only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)
  • the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui
  • the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens
  • if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over
  • due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048
  • all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed
  • added more thumbnail-resizing related popup spam to file report mode
  • fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded
  • updated database migration help for new concepts and ui
  • cleaned up some misc storage code

  • the rest

  • fixed a problem in the client api with fetching file identifiers from file_ids
  • fleshed out 'help my db is broke.txt' with more specific clone recovery examples
  • fixed import support for a variety of single-frame music webms
  • fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set
  • network report mode now states url classes of urls about to be parsed
  • misc small fixes and cleanup

• version 344

  • final v1.0 client api polish

  • added optional 'show_destination_page' arg to '/add_urls/add_url', defaulting to False, to control whether an URL-add will select (i.e. jump to) the destination page in the ui. this changes the default behaviour for this command
  • simplified the routine that finds or creates a watcher or url import page and fixed a bug in the api that was not creating new pages when destination_page_name was specified
  • some misc cleanup
  • fixed fetching file_metadata by hashes
  • fixed the client api help regarding file_metadata response example tags
  • client api version is now 5

  • the rest

  • psd support added! because of this format's potential multi-layer complexity, it will not render natively, but width and height are parsed. it is treated as 'application/x-photoshop'. PSB is also recognised and treated as psd
  • added a 'open_known_url' shortcut to the 'media' shortcut set that lets you quickly open URLs for files. if there is one recognised known url, it will be launched, and if there are multiple, a list with all known urls will appear to select which one you want
  • animation scanbars now show an x/y current timestamp! it includes millisecond timings and even works for variable framerate gifs. whether to show a second/shadow caret for timestamp position on variable frame rate is a new discussion to have
  • fixed an issue where animations would sometimes not resume animation for several seconds after a big scanbar drag
  • when the thumbnail manager cannot produce a thumbnail due to a storage error (like a missing file), it now only puts up a single, more informative error popup on the first problem. subsequent errors are printed silently to the log. (these errors tend to come in en masse, so this cuts down on spam and error-related ui lag that was making loading a bad session difficult)
  • improved error reporting when an upload pending command would fail due to service non-functionality--it should now give a popup with error info imediately, rather than obscured through the login system
  • added temp_dir parameter to the client and server that will override which temporary directory the program will use
  • cleaned up how no_daemons and no_wal mode are handled internally
  • no_wal mode now has to be called from the command parameter, the no_wal file hack in the db directory no longer works
  • missing ffmpeg errors now prompt the user to check if it is installed
  • searching for numerical ratings should now work for files that were rated when the service had a different number of stars (ratings now searches in 'bands' rather than exact values)
  • reduced the min height of the new import files frame's list
  • doubled the decompression bomb test to permit files up to ~179 megapixel, we'll see how it goes
  • misc cleanup

• version 343

  • client api

  • fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working
  • fixed up some last /get_files/search_files stuff
  • added /get_files/file_metadata
  • added /get_files/file
  • added /get_files/thumbnail
  • added help and unit tests to reflect the above
  • updated client api version to 4

  • the rest

  • the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired
  • animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media
  • increased aggression of media viewer image prefetch
  • added support for 'MM' Tiffs
  • fixed webm mime parsing for webms with no audio (these were falling back to mkv)
  • improved the error reports when a serialised png fails to import
  • the mass-open-urls popup is now pausable as well as cancellable
  • fixed several recently broken ui unit tests
  • misc old code cleanup
  • some misc test controller/constant refactoring

• version 342

  • added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame
  • added support for tiff import. it works ok for 24bit and 8bit (monochrome) tiffs, but I am not sure how well it will do with 48bit
  • both webp and tiff should work on the duplicate files system
  • improved webm detection to include opus audio (previously, these files were falling back to mkv)
  • fixed an issue where unusual formats with duration but no frames or frames but no duration were being sorted and otherwise presented incorrectly
  • improved autocomplete job cancelability. this job can now cancel much faster on large jobs, meaning typing searches with large result sets will hit less CPU and return faster on subsequent keystrokes
  • _all_ of the complicated 'copy url' commands from the thumbnail right-click->known urls menu are now available on the 'open' submenu! if there is more than one url to open (e.g. 'open all of these files blahbooru post urls' on a selection of 50 files), you will be presented with a yes/no dialog to confirm, and it will open one url in your browser every second (with a cancellable popup if num_urls > 5)
  • by default, system:everything is now hidden if its total files is >10k. you can force it to always show under options->default system predicates
  • the gallery downloader's list's status column now shows gallery status (deferring to active file status) when appropriate and shows 'done!' when all work is complete
  • after working back and forth with a user, I _believe_ the linux similar files >0 distance search crash is finally fixed
  • fixed sorting by media views/viewtiming with collections
  • a single-selected collection right-click now shows total media views for all files in the collection! you can now see how long you have been viewing an artist!
  • fixed an issue that lead to export folders not running on always-on clients as often as they should
  • updated the gelbooru 0.2.5 file page parser to pull rating tag from the correct location (previously, it was pulling from what appears to be a site-wide 'mature' browser hint)
  • improved memory cleanup stability when animations and other parts of the video rendering pipeline are deleted--this _may_ fix some rare crashes
  • increased animation rendering aggression overall and particularly in 'future' of frame buffer
  • if a video renderer that is asked to start some way into the video fails to render anything, it will now fall back to trying to render from the beginning. this is slightly hacky atm and leads to out of phase rendering frames, but it is better than an error
  • added a '--no_db_temp_files' launch parameter that will force the client or server to return to the recent old behaviour of exclusively using memory for journalling. this is useful if your temp directory is small and/or your available ram is very large. if running in this mode, the client will attempt to check available memory (instead of free space on your temp dir) before performing very large transactions
  • with the new lighter-weight update transactions, the client now tests for less free space for journalling before running repository update processing
  • added /get_files/search_files to the client api, which does the first half of file searching. it allows tag search (including -tag negation) and system inbox/archive. since the second half, which will fetch file metadata, is not yet in, this can't do anything interesting yet
  • updated help and unit tests to support this, client api version is now 3
  • some misc refactoring

• version 341

  • client api

  • added /add_tags/add_tags, which does several kinds of tag content updates
  • added /add_tags/clean_tags, which shows how hydrus will handle potential tags
  • added /add_urls/associate_url, which allows you to associate urls with files
  • added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)
  • updated client api version to 2
  • updated client help and unit tests for the above
  • added a linked contents to the client api help
  • improved some server error handling, mostly moving 403s to more correct 400s
  • improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500

  • the rest

  • tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user
  • when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)
  • added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details
  • reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements
  • the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows
  • when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8
  • improved serialisability of some pending tag data
  • watchers can now hold and pass on fixed pending tag data
  • gallery log objects can now hold and pass on fixed pending tag data
  • file import objects can now hold and action fixed pending tag data
  • hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects
  • improved some url-import page drop-target-selection logic
  • improved error reporting when dropping/api-adding urls
  • adjusted some url import workflow so big 'already in db' download lists should work a bit faster
  • attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made
  • tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth
  • fixed a source of linux CRITICAL logspam related to several common dialogs
  • improved ui stability on boot when file folders are missing (particularly for linux)
  • improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better
  • did some more misc stability improvements, particularly in various boot fail scenarios
  • completely removed an ancient and janky focus catcher widget from the main gui frame
  • now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory--hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large
  • fixed approx bitrate sorting for malformed video files with explicitly '0' duration
  • daemon_profile_mode now spams some more info about export folders
  • fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status
  • updated windows build to cv 4.0
  • misc refactoring and fixes

• version 340

  • client api

  • fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly
  • fixed the 'add_url' api call's instability
  • the API will now always return JSON on 200. anything else should be presumed to be raw text
  • '/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week
  • '/request_access_permissions' now returns JSON
  • '/add_url' now results JSON on success with more info, 403 on failure
  • '/get_url_info' now returns the 'normalised_url' in the response JSON
  • added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url
  • added '/add_files/add_file', which can import a file from a path or bytes
  • added '/add_tags/get_tag_services', which will return info on the client's tag services
  • updated client api help to reflect the above changes and fleshed out the intro a bit
  • fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time
  • updated the client api tests to check the above
  • refactored client api tests to be neater and in their own file

  • the rest

  • fixed the page of pages close bug
  • added a downloader for nijie.info to the client defaults (it needs a login)
  • updated danbooru file page parsers to get 'rating' tag
  • added gelbooru 0.1.11 parser for future application
  • fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction
  • added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)
  • wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest unicode replacement characters
  • the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks
  • moved some more old daemons to the new job scheduler, deleted some old daemon code
  • improved some daemon job wake and shutdown code
  • wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code
  • simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update
  • fixed upnp mapping fetching to cope with ipv6 results
  • improved some memory clearing code to deal with some semi-stubborn objects
  • improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine
  • improved menubar stability, both in finding menus and swapping them out
  • if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup
  • fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions
  • fixed two login invalid cookie error handling bugs
  • maybe made some dupe filter searching more stable
  • fixed a py2 datatype issue that made the client unbootable when updating the client from <296
  • the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go
  • slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB
  • added experimental secret discord bugfix dnd mode checkbox
  • improved how html parsing deals with some unexpected bad tag data
  • turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays
  • wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work
  • improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up--this is an ongoing project
  • improved how some text import line splitting works
  • misc fixes

• version 339

  • client api

  • wrote some ui to handle client api permissions adding and editing
  • wrote a 'catch a permissions request' mini-dialog for external api permissions adding

  • wrote api calls

  • GET
  • /api_version
  • /request_new_permissions
  • /verify_access_key
  • /add_urls/get_url_info
  • POST
  • /add_urls/add_url
  • and made a new 'client api' help page to describe in detail what these do
  • wrote fairly comprehensive unit tests for the new client api
  • refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses
  • neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors
  • deleted very old prototype file/thumbnail client server fetch code, which was no longer in use

  • misc and bug fixes

  • added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or _all_ files
  • the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)
  • the file url downloader now reports 'downloading file' and 'importing url' text status separately
  • fixed a typo bug from last week that was breaking asc/desc ratings service sorting
  • fixed a typo bug from last week that was stopping manage import folders from opening
  • fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel
  • the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)
  • the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension
  • fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters
  • the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)
  • fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well
  • maybe cleaned up some menubar management code
  • the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook
  • fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running
  • improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into
  • improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save
  • improved thread-safety of serialisable objects as they serialise
  • misc improvements and cleanup

• version 338

  • after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate ~asynchronously~, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios _should_ be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc...
  • I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here
  • wildcard search tags now have an explicit '(wildcard search)' after their label
  • taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!
  • added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side
  • some of the file sort options now default to their respective 'biggest' first, see how you like it
  • folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools
  • if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging
  • fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view
  • fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked
  • wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'
  • the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively
  • updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made
  • removed some old experimental crypto code that isn't used any more--pycryptodome is no longer needed to run the program from source
  • improved some misc client service code
  • deleted some old unused code
  • misc cleanup
  • updated to ffmpeg 4.1 on windows release
  • updated to sqlite 3.26 on windows release
  • updated to wxpython 4.0.4 on all platforms

• version 337

  • fixed another couple of unicode encoding problems with the logging and profiling code
  • the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding
  • fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video
  • fixed some 'max size' download file size testing
  • 'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)
  • the downloader easy-importer lain image is now clickable to launch a file selection dialog
  • if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you
  • duplicate content merging now applies to pending as well as current tags
  • the duplicate filter now counts pending tags when saying which file has more tags
  • advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly
  • stopped printing long server error text on 304 and 404, where actual response content is uninteresting
  • removed wx import that accidentally came into server space due to 2to3 check
  • improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds
  • fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else
  • improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements
  • did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations
  • reduced db disk cache aggression significantly and added more memory maintenance to the cache population process
  • fixed an issue where subscriptions were not promptly responding to shutdown events
  • fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events
  • added a 'pubsub report mode' debug mode for simpler pubsub review
  • the db is now less redundantly spammy on certain behind the scenes update notifications
  • wrote a first version of the client api manager and permissions handling objects
  • misc fixes, cleaned some shutdown code

• version 336

  • fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)
  • fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases
  • fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)
  • fixed a str vs bytes issue when loading the filename tagging panel
  • fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)
  • improved some json dump deserialisation code
  • fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel
  • improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data
  • fixed some hex character processing for system preds
  • added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)
  • the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation
  • made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update
  • the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system
  • simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method
  • wrote some new help.txt in the db dir about hanging startups

• version 335

  • important:
  • hydrus now runs completely and exclusively on python 3!

  • for users who are updating, the client has special install instructions for just this week

  • if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)
  • if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)
  • if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!
  • if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out
  • please check the release post for more detailed instructions here

  • semi-important

  • the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from ???? ????? lads appreciated
  • I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)
  • although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source
  • I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries

  • misc bug fixes

  • fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class
  • removed some popup flicker related to long error messages
  • fixed some parsing testing ui error handling
  • cleared up some bad text ctrl event handling that could sometimes cause a recursive loop
  • listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion
  • cleaned up some bad external process calling code and improved how child process environment is set up
  • finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this

  • boring details

  • ran 2to3 to auto-convert what could be done
  • updated environment to python 3
  • went over a whole ton of unicode encoding/decoding manually to update it to python 3
  • removed all the old tobytestring/tounicode calls in favour of new python 3 handling
  • fixed all the file io to do bytes/str as appropriate
  • corrected a bunch of / vs // int/float stuff
  • fixed up twisted, which has some str/bytes stuff going on
  • fixed all the listctrls to deal with column sorting None values amongst ints/strs
  • fixed png export/import, which had some fun bytes/bytearray/int/str issues
  • updated the swf header parsing code to py3 (more str/bytes stuff)
  • misc float/int fixes
  • fixed up some http.cookies handling, which has changed in py3
  • improved some ancient loopback connection code that was only really checking to see if ports were in use
  • cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked
  • numerous misc other refactoring and so on
  • updated the new network engine to now decode non-utf-8 responses correctly based on actual response header
  • removed some old py2 manual http multipart code
  • removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection
  • replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess
  • deleted some more misc old code
  • tightened up some spammy network error reporting
  • converted all /r/n to /n in my environment project, ha ha ha
  • the ui seems to better support rarer unicode characters like ??
  • updated some of the install/update/backup help for all this, and some misc other stuff as well
  • fixed misc bugs

• version 334

  • wrote a system:file viewing stats to comprehensively search the new viewing stats--it _should_ also be synced with the exact current values
  • but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!
  • added a _database->file viewing stats_ menu that lets you suspend file view tracking and clear all records permanently
  • mr. bones now welcomes all users under the help menu
  • fixed mr. bones's confusion at those who have yet to board the ride
  • also mr. bones now makes sure to get the latest file viewing stats
  • moved confirm trash/archive from _options->gui_ to _options->files and trash_
  • moved a bunch of 'pages' related stuff from _options->gui_ to the new _options->gui pages_
  • added an option to options->gui pages to change the number of session rolling backups
  • subscription popups now provide an x/y query progress string in their popup text
  • the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default
  • fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import
  • the client will now print up to 512KB of server error info to the log (previously 4KB)
  • removed the youtube download prototype--if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client
  • network report mode now shows more network error information
  • gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience
  • wrote a 'logins' section to the bottom of the 'getting started with downloading' help page
  • misc fixes

• version 333

  • added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!
  • you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media
  • viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)
  • you can now sort files by total media views/viewtime!
  • mr. bones's wild ride continues, as well
  • deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!
  • tag siblings refresh is quicker and less memory heavy thanks to this as well
  • furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results
  • the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!
  • tag and rating sort options are now listed as 'tag:' and 'rating:' respectively
  • fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations
  • fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist
  • the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist
  • cleaned up some of the hover_window-canvas interaction code
  • fixed some long-time sperg-out buffer-drawing when changing position in a long video
  • the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate
  • fixed some ancient repository admin code that fetches summary account info given an account key
  • the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors
  • fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation

• version 332

  • the client serialisation system now supports multiple rolling backups!
  • client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!
  • when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations
  • gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload
  • tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch
  • under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively
  • attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process
  • did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system
  • wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster
  • export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.
  • fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work
  • when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!
  • the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads
  • adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well
  • fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices
  • fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add
  • the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it
  • wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places
  • improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service
  • fixed a small typo bug when trying to auto-add url classes and parsers
  • updated an old deprecated checkboxlist call
  • clarified the integrity section of 'help my db is broke.txt'
  • misc fixes

• version 331

  • added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal
  • 'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies
  • the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this
  • login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)
  • brushed up some of the 'change login script' code--particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection
  • after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong
  • prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes
  • added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!
  • updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org _should_ all be updated right. exhentai.org is likely too difficult to properly support in the current system
  • the different panels where you enter system predicate information now all run on the new panel sizing system--if you have had problems with these, please let me know how they size now!
  • added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens
  • the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files--presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.
  • misc ui improvements
  • updated 'running from source' in help

• version 330

  • login

  • added a proper username/password login script for hentai foundry--double-check your hf filters are set how you want in your profile, and your hydrus should inherit the same rules
  • fixed the gelbooru login script from last week, which typoed safebooru.com instead of .org
  • fixed the pixiv login 'link' to correctly say nsfw rather than everything, which wasn't going through last week right
  • improved the pixiv file page api parser to veto on 'could not access nsfw due to not logged in' status, although in further testing, this state seems to be rarer than previously/completely gone
  • added login scripts from the github for shimmie, sankaku, and e-hentai--thanks to Cuddlebear and any other users who helped put these together
  • added safebooru.donmai.us to danbooru login
  • improved the deviant art file page parser to get the 'full' embedded image link at higher preference than the standard embed, and only get the 'download' button if it looks like an image (hence, deviant art should stop getting 140MB brush zips!)
  • the manage logins panel now says when a login is expected to expire
  • the manage logins dialog now has a 'scrub invalidity' button to 'try again' a login that broke due to server error or similar
  • entering blank/invalid credentials is now permitted in the manage logins panel, and if entered on an 'active' domain, it will additionally deactivate it automatically
  • the manage logins panel is better at figuring out and updating validity after changes
  • the 'required cookies' in login scripts and steps now use string match names! hence, dynamically named cookies can now be checked! all existing checks are updated to fixed-string string matches
  • improved some cookie lookup code
  • improved some login manager script-updating code
  • deleted all the old legacy login code
  • misc login ui cleanup and fixes

  • other

  • sped up tag searches in certain situations (usually huge inbox) by using a different optimisation
  • increased the repository mappings processing chunk size from 1k to 50k, which greatly increases processing in certain situations. let's see how it goes for different users--I may revisit the pipeline here to make it more flexible for faster and slower hard drives
  • many of the 'select from a list of texts' dialogs--such as when you select a gallery to download from--are now on the new panel system. the list will grow and shrink depending on its length and available screen real estate

  • misc

  • extended my new dialog panel code so it can ask a question before an OK happens
  • fixed an issue with scanning through videos that have non-integer frame-counts due to previous misparsing
  • fixed a issue where file import objects that have been removed from the list but were still lingering on the list ui were not rendering their (invalid) index correctly
  • when export folders fail to do their work, the error is now presented in a better way and all export folders are paused
  • fixed an issue where the export files dialog could not boot if the most previous export phrase was invalid
  • the duplicate filter page now has a button to more easily edit the default merge options
  • increased the sibling/parent refresh delay for 1s to 8s
  • hydrus repository sync fails due to network login issues or manual network user cancel will now be caught properly and a reasonable delay added
  • additional errors on repository sync will cause a reasonable delay on future work but still elevate the error
  • converted import folder management ui to the new panel system
  • refactored import folder ui code to ClientGUIImport.py
  • converted export folder management ui to the new panel system
  • refactored export folder ui code to the new ClientGUIExport.py
  • refactored manual file export ui code to ClientGUIExport.py
  • deleted some very old imageboard dumping management code
  • deleted some very old contact management code
  • did a little prep work for some 'show background image behind thumbs', including the start of a bitmap manager. I'll give it another go later

• version 329

  • login

  • the login manager is fully turned on! hentai-foundry click-through and pixiv login now occur fully on the new system
  • wrote a Deviant Art login script for NSFW downloading--however, it only seems to work on a client that has done some logged-out downloading first (otherwise it thinks you are a robot)
  • updated the DA file page parser to only NSFW-veto if the user is currently logged out
  • wrote a danbooru login script for user prefs and special files if you have a gold account
  • wrote a gelbooru 0.2.x login script for user prefs
  • pixiv recently(?) allowed non-logged in users to see sfw content, so the login script is updated to reflect this. the login script doesn't detect a failed login any more, so I will revisit this
  • logging in in the regular order of things now makes a temporary popup message with the overall login status and final result. ~it is cancellable~--and if cancelled, future login attempts will be delayed
  • logging in in the regular order of things now prints simple started/result lines to the log
  • deleted old network->login menu and related code such as the custom pixiv login management. gdpr click-through is now under downloaders
  • subscription login errors will now specify the given login failure reason
  • subscription login tests will now occur at a better time, guaranteeing the sub will be correctly saved paused if the test fails
  • login errors will now always specify the domain for which they failed
  • testing a login script on a fresh edit login script dialog now pre-fills the alphabetically first example domain
  • the login script test ui now restores its 'run test' button correctly if the test is abandoned early
  • misc improvements to login error handling and reporting

  • other

  • any texts across the program that ellipsize when they are too thin to display what they have will now tooltip their text (this most importantly includes the status on the network job control, which will now display full login problem info)
  • the copy button on manage tags goes back to copying all if no tags are selected
  • the remove button on manage tags now removes only selected if some tags are selected. it still removes all if none are selected
  • the remove button on manage tags is now wrapped in a yes/no dialog (as is hitting the delete key on the list's selection). this can be turned off under the cog button
  • filename tagging panels now support directory tagging for the last, second last, and third last directories. the related code for handling directory tagging is cleaned up significantly
  • the export files panel now lets you delete the files from the client after export. this value will be remembered, and if on will prompt a capital letters warning on export, either via the button or the quick-export shortcut
  • in manage tag parents, where there are multiple parents in a pending action (either by importing via clipboard/file or by putting multiple parents in right-hand box), the action will now be treated as one transaction with one 'enter a reason' confirmation!
  • in manage tag siblings, when multiple 'better' values are pended in one action via a clipboard/file import, they will now be treated as one transaction with one 'enter a reason' confirmation!

  • misc

  • added a new url class that api-links .gifv-style imgur links so they are downloadable like regular imgur single media pages
  • the pixiv manga page url class now redirects to the new api, so mode=manga pages should now be drag-and-drop importable and generally downloadable if you have any still hanging around in any queues
  • clients now come with an additional danbooru parser that fetches the webm version of ugoiras
  • after discovering a pdf that ate indefinite 100% CPU while trying to parse, I have decided to stop pulling num_words for pdfs. it was always a super inaccurate number, so let's wait for a better solution at a later date. hydrus hence no longer requires pypdf2
  • fixed an issue with monthly bandwidth estimates rolling over to the new year incorrectly
  • in an attempt to chase down a duplicate files content move/copy bug, the duplicate action content updates got a bit of cleanup work. if you have noticed duplicate actions not copying tags/urls, please let me know the exact process in the ui, including services and merge options, you went through
  • tag lists should now update their sibling appearance correctly after a tag siblings dialog ok--previously, they were checking for new sibs too early
  • tag siblings and parents should now refresh their data more efficiently when spammed with new data notifications (this usually happens janitor-side, which approving dozens at once)
  • copy queries/watcher urls on the download pages' lists' right-click menus no longer double-spaces the copied texts (it just does single spaces)
  • fixed an issue where certain initialised watchers were erroring out when asked to provide next-check time estimates--in all cases, null timestamps will be dealt with better here
  • misc tag parents/siblings ui code cleanup
  • wrote some code to catch and report on an unusual dialog dismissal error

• version 328

  • wrote test ui for edit login script panel
  • the login system now works and is turned on, although the legacy hardcoded pixiv and hf logins remain in place. it will not do anything very new this week--it is strictly only for advanced users to experiment with for now
  • cleaned up some messy network code
  • all subscription and hydrus jobs will no longer wait indefinitely on an invalid login--they will cancel immediately
  • network jobs will report a bit more info when they are cancelled
  • subscriptions will now attempt to test login validity before and during file downloads and syncing. if they fail, the sub will pause and stop work and a message will be presented to the user
  • made a 'thumbnails' options page and moved some things to it
  • added thumbnail border and margin to that thumbnails page! you can even set 0 border and/or margin and it works
  • fixed up a heap of bad thumbnail drawing code that didn't work with thicker borders
  • the tag sibling and parent dialogs now have suggestion buttons in their 'give a reason' dialogs! if your petitions are simple and fit into one of these categories (which is most of them), please use these buttons as they will let janitors (e.g. hydrus dev for the PTR) process them in batches, in fewer clicks
  • manage tags dialog's checkboxes and advanced buttons are now wrapped into a cog icon! remove/copy/paste buttons are compacted and put on the same row!
  • manage tags dialog's copy button now only copies selected, not all tags
  • manage tags dialog now uses the new sizer. some components are smaller by default but will eat up spare pixels better
  • misc manage tags code cleanup
  • added 'paged file import queues' to the network->pause menu. this will pause any, hdd, url, simple, gallery, or watcher page from processing its file import queue. it is a bit hacky and will take up to 30s to unpause unless you joggle the respective downloader to wake it up--see how it goes!
  • added a similar 'gallery searching' to that menu, to pause any url, simple, or gallery page from actioning its gallery-side queue
  • and 'watcher checking' for watchers
  • fixed a stupid bug that was causing false-positive and _mostly_ harmless errors for certain pixiv and artstation multi-page downloads
  • fixed an issue where multi-page data was being mis-metadata'd (mostly, this meant thread watchers were giving the last filename tag to all files, and I think similarly getting the latest source time) due to a mistake in some recent de-duping code
  • fixed the new pixiv file page api parser to parse source time, which I must have accidentally deleted previously
  • fixed a no-expand bug in my new sizer when in horizontal orientation
  • fixed a small bug when making an easy-import downloader png and cancelling the add url class dialog
  • misc fixes

• version 327

  • login stuff

  • finished off some login script data stuff
  • fleshed out how login credentials and other linked data is stored in the login manager, including script link recovery when the script changes but name does not
  • improved some initialisation login validation error handling
  • improved login failure validation error handling
  • wrote a dialog panel for managing login credentials and reviewing validity and so on
  • a heap of related session and login tie-in/fix-up work
  • the login manager will now save changes to the db. it will get the HF and pixiv scripts on db creation/update, and if you have a pixiv login, the login system will pre-fill that info and 'activate' the script (although the login manager will not fire any login scripts yet--if so configured, it'll just delay on a polite error message)

  • other stuff

  • with the subscriptions' new more liberal syncing logic, the periodic file limit will now only pop up if the sub does not see any already-seen files
  • to give more buffer for the new syncing logic, file import caches will now store 250 entries minimum on compaction (was 100 previously)
  • subscription merging now lets you choose the primary subscription into which the other subs will be merged
  • cancelling a subscription merge action mid-merge is now safely nullipotent
  • post urls that use subsidiary page parsers (such as the new pixiv manga parser) will now correctly insert (rather than append) their manga urls into the file import cache
  • removed a couple of places where urls could accidentally be duplicated in a file import cache
  • cleaned up some areas where successful file import objects were presumed to have file hashes when they might not (this was causing errors when importing urls that split into multiple url children, like pixiv manga, while also having 'additional tags' set)
  • updated tag censorship, parents, and siblings dialogs to the new panel system
  • tag censorship, parents, and siblings panels now use a notebook instead of the layout-borked listbook
  • tag parents and siblings panels now use the new small-resolution-friendly sizer, are more tight by default, and expand more neatly
  • refactored a bunch of tag ui code to clientguitags
  • the client video renderer will now deal with videos with (invalid) duration of 0 more gracefully
  • finished the 'getting started with downloading' help page, sans the login stuff
  • bit of other help work

• version 326

  • login

  • finished the new login objects. they can deal with multi-step single second-level domain login problems, can pass variables from step to step, and use cookies as success verification
  • wrote an ton of ui for the new login objects, now under network->downloader definitions and network->logins. it is not 'active' yet, but advanced users are invited to check it out. there is no good test ui yet, which I think I'll have to figure out in the coming weeks
  • wrote a first attempt at HF and pixiv replacement login scripts--please try importing from defaults on the manage login scripts dialog and look through them to see what I am going for. once the system is flipped on and we are happy these work, I'll remove the old hardcoded legacy login stuff
  • when a network job that needs a login cannot login, it now waits (rather than bombing out completely), presenting the related error, and checks again every 60 seconds
  • if a network job thinks it can login but fails to generate a login process, the network engine now catches the error safely and recovers. the job is put on hold as above

  • subs

  • the subscription 'have we caught up to where we were before' test is now more complicated--rather than just stopping after five 'already seen' urls are found, it now only stops if at least the _last_ five contiguous urls of the page are already seen. this will catch more late-tagged files that get inserted out of order
  • fixed the 'get quality info' button on edit sub panel to only get the current selection, not all queries wew
  • subscriptions can now optionally publish/present their files to a specific label! this is a great way to merge multiple subs to the same final landing page

  • layout

  • after a long time thinking about it, wrote a new custom boxsizer that handles resizing multiple expanding items of different reasonable min size by expanding them _beyond minimum size_ by their proportion, rather than forcing them all to have total proportional width/height. I expect to polish this and apply it in multiple locations around the program where tall things were being too tall because something else was forcing it to be (the management panel on the left of most pages was terrible at this, causing a giganto taglist just because the upper panel was tall as well).
  • changed my custom boxsizer (the box with a bold header) to the new custom boxsizer, so it is all over now--please report any bad layouts you see
  • in an effort to improve layout, the manage tag parents and siblings panels' preview boxes have shorter minimum height--it will get a bigger layout overhaul soon

  • bigger misc

  • thanks to work of user kourraxspam on the discord, fixed the pixiv downloader to use a more stable api and added pixiv tag search
  • watchers and gallery imports now have a list right-click menu entry to show all selected importers' files in a new page! use this to clear out a bunch of finished queues all at once!
  • the tag right-click menu now offers 'open new search pages for each in selection' if multiple tags are selected--this will open three search pages each with one tag, as opposed to the original entry, which would only open one page with all three
  • the edit nested gug panel now uses a checklistbox rather than the menu to select gugs to add, which is more reliable and allows for multiple selections
  • sped up autocomplete tag fetches' tag sibling integration--irl this may be a reduction in total a/c search time of approx 33%
  • page parsers will now generate next gallery urls absent any file/post urls if the only type of url they can generate is gallery urls (so a meta-gallery-search like board->threads that only generates subsidiary gallery pages will now work, whereas before it never could because it was missing post urls)
  • the gallery log now provides a shorthand way to restart and resume failed searches from its right-click menu (if the most recent log entry failed)
  • 'try again (and allow search to continue)' reattempt jobs will now generate next page urls even if no new urls are found (which can happen if a search stopped due to the file limit exactly lining up with the number of files found, for instance, so a reattempt finds nothing new)
  • gallery downloaders will now specify their 'delay work for a bit' error states in the ui. this usually means 'could not connect', which has a 4-hour timer (I'll prob add a scrub delays button here at some point)
  • the watcher will now show its 'delay work for a bit' error state in more places in the ui
  • added a 'media' shortcut 'export_files_quick_auto_export', which will open the export files frame and give you a quick yes/no to confirm you want to export as set. if yes, it will export. then it will close the frame
  • added a 'show a "N" to short import summaries' option to options->downloading, which will extend the typical 'x/y' status string to 'x/y - zN' for z 'new files' (as opposed to already in db)
  • improved how the video parser estimates frame rate--it _should_ fix some of those low-framerate, low-framecount slideshow-vids where at current they render everything in a rush and then sit on the last frame for ten secs

  • smaller misc

  • network report mode now reports url_to_fetch and parser-to-parse-with info
  • when the server fails to accept a file upload due to a file parsing issue, it now prints the hash of the file in the error
  • if the client sees a possible file hash in a server error message from a file upload, ~it will try to show that file in a new page~
  • fixed an issue where wildcard searches were not finding results if the search text included the normally discarded characters [](){}"'
  • fixed some domain handling for localhost and other undotted network names
  • content parsers will now only launch with permissable content types, which for the legacy 'lookup scripts' scripts system means only tags and vetoes, and for the new login system means only temp variables and vetoes
  • as compaction now happens automatically on sync, removed the 'compact' button from edit subs panel
  • an unusual network error related to hydrus update files sometimes being cut off mid-stream is now glossed over silently, with the download reattempted after a delay
  • the initial gui session load now occurs after a 0.25s delay--let's see if it cleans up some initial layout issues some users have had
  • maybe fixed an odd dictionary-initialisation error related to tag siblings/parents dialog boot
  • ruggedised against an unusual bandwidth load bug
  • gave some of the index help a pass
  • did most of a 'getting started with downloaders' page in the help--I'll finish it next week
  • updated discord share link to https://discord.gg/3H8UTpb , which should not expire
  • some listbox add/edit code cleanup
  • some listctrl delete code cleanup
  • misc help work
  • misc cleanup

• version 325

  • added a 'show a popup while working' checkbox to edit subscription panel--be careful with it, I think maybe only turn it off after you are happy everything is set up right and the sub has run once
  • advanced mode users will see a new 'get quality info' button on the edit subscription panel. this will some ugly+hacky inbox/archived/deleted info on the selected queries to help you figure out if you are only archiving, say, 2% of one query. this is a quickly made but cpu-expensive way of calculating this info. I can obviously expand it in future, so I would appreciate your thoughts
  • subscription queries now have an optional display name, which has no bearing on their function but if set will appear instead of query text in various presentation contexts (this is useful, for instance, if the downloader query text deals in something unhelpful like integer artist_id)
  • subscription queries now each have a simple tag import options! this only allows 'additional tags', in case you want to add some simple per-query tags
  • selecting 'try again' on file imports that previously failed due to 'deleted' will now pop up a little yes/no asking if you would like to first erase these files' previously deleted file record!
  • the watcher and gallery import panels now have 'retry failed' buttons and right-click menu entries when appropriate
  • the watcher and gallery import panels will now do some ui update less frequently when they contain a lot of data
  • fixed the new human-friendly tag sorting code for ungrouped lexicographic sort orders, where it was accidentally grouping by namespace
  • downloader easy-import pngs can now hold custom header and bandwidth rules metadata! this info, if explicitly present for the appropriate domain, will be added automatically on the export side as you add gugs. it can also be bundled separately after manually typing a domain to add. on the import side, it is now listed as a new type. longer human-friendly descriptions of all bandwidth and header information being bundled will be displayed during the export and import processes, just as an additional check
  • for advanced users, added 'do not skip downloading because of known urls/hashes' options to downloader file import options. these checkboxes work like the tag import options ones--ignoring known urls and hashes to force downloads. they are advanced and should not be used unless you have a particular problem to fix
  • improved how the pre-import url/hash checking code is compared for the tag and file import options, particularly on the hash side
  • for advanced users, added 'associate additional source urls' to downloader file import options, which governs whether a site's given 'source urls' should be added and trusted for downloaded files. turn this off if the site is giving bad source urls
  • fixed an unusual problem where gallery searches with search terms that included the search separator (like '6+girls skirt', with a separator of '+') were being overzealously de/encoded (to '6+girls+skirt' rather than '6%2bgirls+skirt')
  • improved how unicode quoted characters in URLs' query parameters, like %E5%B0%BB%E7%A5%9E%E6%A7%98 are auto-converted to something prettier when the user sees them
  • the client now tests if 'already in db' results are actually backed by the file structure--now, if a the actual file is missing despite the db record, the import will be force-attempted and the file structure hopefully healed
  • gallery url jobs will no longer spawn new 'next page' urls if the job yielded 0 _new_ (rather than _total_) file urls (so we should have fixed loops fetching the same x 'already in file import cache' results due to the gallery just passing the same results for n+1 page fetches)
  • in the edit parsing panels, if the example data currently looks like json, new content parsers will spawn with json formulae, otherwise they will get html formulae
  • fixed an issue with the default twitter tweet parser pulling the wrong month for source time
  • added a simple 'media load report mode' to the help debug menu to help figure out some PIL/OpenCV load order stuff
  • the 'missing locations recovery' dialog that spawns on boot if file locations are missing now uses the new listctrl, so is thankfully sortable! it also works better behind the scenes
  • this dialog now also has an 'add a possibly correct location' button, which will scan the given directory for the correct prefixes and automatically fill in the list for you
  • fixed some of the new import folder error reporting
  • misc code cleanup

• version 324

  • downloaders

  • after adding some small new parser tools, wrote a new pixiv downloader that should work with their new dynamic gallery's api. it fetches all an artist's work in one page. some existing pixiv download components will be renamed and detached from your existing subs and downloaders. your existing subs may switch over to the correct pixiv downloader automatically, or you may need to manually set them (you'll get a popup to remind you).
  • wrote a twitter username lookup downloader. it should skip retweets. it is a bit hacky, so it may collapse if they change something small with their internal javascript api. it fetches 19-20 tweets per 'page', so if the account has 20 rts in a row, it'll likely stop searching there. also, afaik, twitter browsing only works back 3200 tweets or so. I recommend proceeding slowly.
  • added a simple gelbooru 0.1.11 file page parser to the defaults. it won't link to anything by default, but it is there if you want to put together some booru.org stuff
  • you can now set your default/favourite download source under options->downloading

  • misc

  • the 'do idle work on shutdown' system will now only ask/run once per x time units (including if you say no to the ask dialog). x is one day by default, but can be set in 'maintenance and processing'
  • added 'max jobs' and 'max jobs per domain' to options->connection. defaults remain 15 and 3
  • the colour selection buttons across the program now have a right-click menu to import/export #FF0000 hex codes from/to the clipboard
  • tag namespace colours and namespace rendering options are moved from 'colours' and 'tags' options pages to 'tag summaries', which is renamed to 'tag presentation'
  • the Lain import dropper now supports pngs with single gugs, url classes, or parsers--not just fully packaged downloaders
  • fixed an issue where trying to remove a selection of files from the duplicate system (through the advanced duplicates menu) would only apply to the first pair of files
  • improved some error reporting related to too-long filenames on import
  • improved error handling for the folder-scanning stage in import folders--now, when it runs into an error, it will preserve its details better, notify the user better, and safely auto-pause the import folder
  • png export auto-filenames will now be sanitized of \, /, :, *-type OS-path-invalid characters as appropriate as the dialog loads
  • the 'loading subs' popup message should appear more reliably (after 1s delay) if the first subs are big and loading slow
  • fixed the 'fullscreen switch' hover window button for the duplicate filter
  • deleted some old hydrus session management code and db table
  • some other things that I lost track of. I think it was mostly some little dialog fixes :/

  • advanced downloader stuff

  • the test panel on pageparser edit panels now has a 'post pre-parsing conversion' notebook page that shows the given example data after the pre-parsing conversion has occurred, including error information if it failed. it has a summary size/guessed type description and copy and refresh buttons.
  • the 'raw data' copy/fetch/paste buttons and description are moved down to the raw data page
  • the pageparser now passes up this post-conversion example data to sub-objects, so they now start with the correctly converted example data
  • the subsidiarypageparser edit panel now also has a notebook page, also with brief description and copy/refresh buttons, that summarises the raw separated data
  • the subsidiary page parser now passes up the first post to its sub-objects, so they now start with a single post's example data
  • content parsers can now sort the strings their formulae get back. you can sort strict lexicographic or the new human-friendly sort that does numbers properly, and of course you can go ascending or descending--if you can get the ids of what you want but they are in the wrong order, you can now easily fix it!
  • some json dict parsing code now iterates through dict keys lexicographically ascending by default. unfortunately, due to how the python json parser I use works, there isn't a way to process dict items in the original order
  • the json parsing formula now uses a string match when searching for dictionary keys, so you can now match multiple keys here (as in the pixiv illusts|manga fix). existing dictionary key look-ups will be converted to 'fixed' string matches
  • the json parsing formula can now get the content type 'dictionary keys', which will fetch all the text keys in the dictionary/Object, if the api designer happens to have put useful data in there, wew
  • formulae now remove newlines from their parsed texts before they are sent to the StringMatch! so, if you are grabbing some multi-line html and want to test for 'Posted: ' somewhere in that mess, it is now easy.

• version 323

  • wrote first version of the new downloader easy-import drop-panel. you drop downloader-encoded pngs on it, and it maybe asks you a question and jumbles its way through auto-importing all the required data to the client
  • extended this file import to do some cleverer 'example url merging' when parsers are otherwise dupes, rather than spamming similar dupes on import
  • wrote first version of the new downloader export panel. it takes gugs, url classes and parsers, and predicts sensible sub-objects to include to make functional downloaders, and bundles it into one png
  • fleshed out help for the new easy import/export system
  • the client now slows down gallery and watcher processing when the network engine is under heavy load, aiming for no more than 50 jobs in system at once. the solution is a bit hacky for now, but it should alleviate the deadlock issue when there are ~180+ simultaneous gallery/watcher network jobs pending
  • the multi-watcher panel's list of watchers now supports right-click menu to copy/open urls and pause/play files/checking
  • the multi-downloader panel's list of downloaders now supports right-click menu to copy query texts and pause/play files/searching
  • added a 'derpibooru tag search - no filter' GUG that disables the default derpi no-explicit-files rule
  • added basic gfycat support to default client--drag and drop any typical video page, and it should import ok
  • fixed the canvas/hover window tag sorting discrepancy--all tags are now sorted with the same code, and the media view sort order should be the same as your default sort order (although in this case incidence has no effect as there are no tag counts)
  • rewrote the network job control's cog menu to be a bit more dynamic, and added 'override gallery slot requirements for this job' if appropriate
  • fixed a stupid typo bug in the shutdown maintenance jobs test code that was causing pending repository work to not report right
  • fixed gallery searches that include unicode characters that end up in the path of the url (rather than the query parameters)
  • fixed an issue where highlighting a watcher would unpause its checking
  • generalised the way the new listctrl class can produce right-click menus
  • fixed some api link calculation that was over-prescribing api link display pairs (this affected the artstation file page url class by default). these pairs are now also sorted in the links dialog
  • misc png-export improvements to present better with the new easy import/export stuff
  • the summary texts in the tag filter panel now ellipsize (...), so if the tag filter is complicated, it won't try to boot a superwide edit panel!
  • the manage subscriptions panel now correctly initially sorts in a case-insensitive way (previously, it was usually sorting A-Za-z, which is different to regular aA-zZ resorting behaviour, so it always sort-flickered after the first edit)
  • the status bar has a new segment for reporting when the client is 'busy' with different jobs. for most typical usage, it'll just stay blank. let's see how it goes.
  • fixed mr. bones's wild review when the client currently has no files
  • punched up the new file report mode to specify full paths where available
  • improved some misc downloader code

• version 322

  • wrote gugs help
  • gave url classes and parsers help a pass
  • wrote e621 html gallery page example help
  • wrote gelbooru html file page example help
  • wrote artstation json file page example help
  • wrote url class links help
  • gallery logs for the gallery downloader and url downloader now support 'try again' and 'skip' right-click menu for gallery log entries. the try again allows just the one page or also allowing search to continue) so, if a gallery query fails for some reason, you can now try again/continue where it broke. subs/watcher/simple downloader work on more complicated gallery search logic, so their gallery logs will remain read-only for now
  • all gallery log buttons now support right-click menu to mass-export urls to png or clipboard. non read-only also support import
  • fixed an issue with gallery searches that rely on both api url conversions and url class next gallery page urls (I think just artstation and tumblr by default) not generating the next page url correctly
  • improved some misc gallery url processing logic
  • fixed some issues with gallery url generators with invalid example urls causing problems opening the edit gug and gallery selector panels
  • fixed an issue where you could only delete a gug if it was in an ngug, ha ha
  • thanks to a different submission by prkc on the discord, collections now have a _right-click->set collections as groups of alternates_ duplicate action (note the duplicate menu only appears in advanced mode). the related shortcut action duplicate_media_set_alternate_collections is also added
  • export phrases now support '\' ('/' in linux) in the path export phrase in order to create folders. you should also be able to do \[series]\ to create optional namespace folders. slashes in tags will still be replaced with _
  • to stop the client sometimes doing laggy vacuum checks every maintenance cycle, vacuums that cannot occur due to limited disk space now will still count as 'done' for the purposes of rescheduling
  • added 'file report mode' to the help debug menu. This will spam popups as file and thumbnail actions are requested
  • tightened up some network job status setting to help us debug the 'there are a ton of jobs in network engine, but the three active on this domain seem stalled' issue
  • wrote a simple 'review threads' panel under help->debug->data actions->review threads. I knocked it together in about ten minutes, and it's likely unstable as hell, but it's pretty neat!
  • some instances where many file paths are copied quickly (exporting paths to clipboard and drag and drop) no longer do a safety check for file existence, so should be much faster to go. this particularly reduces startup lag for large file drag and drops!
  • the 'would you like to do maintenance work in this shutdown?' dialog now lists a summary of what it thinks it'll be working on. I _could_ make this more detailed, so let me know how it works for you
  • tags with numbers should now sort according to the new improved human sorting method--it now shouldn't matter where the numbers are in the tag--as long as the text-and-number-breaks lines up with another tag, they'll be compared each part in turn correctly
  • fixed some human sorting code for unusual number characters like ?. they will be treated as text, not a number, for now
  • misc fixes

• version 321

  • downloader overhaul

  • the basic downloader overhaul is complete! at this point, any user can create and share the objects required for a completely new downloader! it is still rough in some places, so a round of EZ-import is coming to make adding new downloaders a single easy drag and drop action
  • rounded out the ngug (nested gugs, which contain multiple gugs) code
  • updated the edit gug panel to deal with gugs and ngugs on different notebook pages
  • added a bunch of logic to this panel and backend data handling to deal with missing gugs in ngugs
  • if an ngug cannot find a gug by its internal identifier key, it will now attempt to fallback to its simple name, and will silently fail if no gug can be found. all gug tracking now uses this 'key first, name later' id method, so downloaders and subs should generally survive gug renames and same-name overwrites
  • the gallery selector now works in gugs. it has two 'pages', depending on which gugs are set to 'display', and will note if the chosen gug is cannot be found in the current definitions. the gugs have slightly more specific names ('gelbooru tag search', 'hentai foundry artist lookup', etc...) than before
  • the gallery selector also puts 'non-functional' gugs (i.e. those with no parsable gallery url class) to a third page
  • moved the gallery downloader gallery and file pipeline completely over to the new system
  • the gallery downloader will now bundle nested gugs (like hentai foundry artist, which searches both works and scraps) into a single downloader
  • moved the subs gallery and file pipeline comppletely over to the new system
  • the subs gallery sync now handles nested gugs (like hentai foundry artist, which searches both works and scraps) in an interleaved manner and make behind-the-scenes checking decisions in a clearer and more logical way
  • subs should now make correct 'hit limit' stop reason reports and not generate new gallery pages when the current page has exactly enough results to hit the current file limit
  • artstation artist lookup is now available as a default downloader
  • newgrounds artist lookup makes a triumphant return. it works pretty well, given how flash and NG has changed since
  • derpibooru tag lookup is now available as a default downloader. due to unusual search syntax on derpibooru, please enter queries exactly as you would on derpi, using ',' or ' AND ' to separate tags (such as 'rainbow dash,straight')
  • pixiv now has multiple artist lookup options--either images, manga, ugoira (doesn't work yet!), or everything
  • the old downloader code is deleted!
  • the old manage booru dialog is deleted!
  • 'custom' boorus (i.e. new ones you created or imported to 'manage boorus'), cannot be completely automatically updated to the new system. I've figured out a way to generate new gugs and gallery&post parsers, but they will miss url classes to get working again. your custom-booru subs will notice this and safely pause until the issue is fixed. if you rely on custom boorus, please check the release post for info on this--you might like to put off updating
  • many misc changes and fixes to gugs and overall gallery url handling pipeline
  • some misc refactoring and concept-renaming in gallery pipeline r.e. gugs
  • when the downloader tries to import what looks like a raw html file, its error notes will specify this and suggest a parser may be needed
  • moved the 'media viewer url display' options panel from the manage url match links dialog to the new network->downloaders->manage downloader and url display
  • this new dialog also hosts a list for managing which downloaders to show in the first list of the downloader selector

  • misc

  • gave the video rendering pipeline communication logic a quick pass, cleaning up a bunch of bad code and other decisions. the video renderer should be quicker to respond to various changes in scanbar position, and incidences of the frame buffer suddenly sperging out (usually inexplicably falling behind the current frame position or deciding to regen for no apparent reason) should be greatly reduced if not completely eliminated
  • the test that stops repository processing if there is not enough disk space now uses half the current size of client.mappings.db for its estimate (previously 1GB) and also tests temp folder location free space (just as the vacuum test does) and reports this nature of the error along with pausing the repo, stopping further attempts
  • might have fixed another out-of-order dialog close/open event combination during manage tags close->advanced content update open
  • fixed gallery queries that include '/' (or some other unusual characters) that end up in the 'path' of the url (as opposed to the query). this fixes 'male/female' on e621, for instance
  • 'advanced mode' users now have a 'nudge subs awake' menu entry below 'manage subs'. this simply wakes the subs daemon (which usually only checks once every four hours), in case any subs are due
  • 'db report mode' now reports every db job as it comes in (formerly, it only reported some optimisation esoterica). this makes it a more lightweight version of 'db profile mode' for several debugging tasks
  • fixed a tiny issue in fetching the 'how boned am I?' stats when the user had zero inbox/everything count
  • fixed a typo in the default new url class object that was breaking the edit ui panel
  • highlighted the quiet filename tagging options on edit import folder panel

• version 320

  • clients should now have objects for all default downloaders. everything should be prepped for the big switchover

  • wrote gallery url generators for all the default downloaders and a couple more as well
  • wrote a gallery parser for deviant art--it also comes with an update to the DA url class because the meta 'next page' link on DA gallery pages is invalid wew!
  • wrote a gallery parser for hentai foundry, inkbunny, rule34hentai, moebooru (konachan, sakugabooru, yande.re), artstation, newgrounds, and pixiv artist galleries (static html)
  • added a gallery parser for sankaku
  • the artstation post url parser no longer fetches cover images
  • url classes can now support 'default' values for path components and query parameters! so, if your url might be missing a page=1 initialsation value due to user drag-and-drop, you can auto-add it in the normalisation step!
  • if the entered default does not match the rules of the component or parameter, it will be cleared back to none!
  • all appropriate default gallery url classes (which is most) now have these default values. all default gallery url classes will be overwritten on db update
  • three test 'search initialisation' url classes that attempted to fix this problem a different way will be deleted on update, if present
  • updated some other url classes
  • when checking source urls during the pre-download import status check, the client will now distrust parsed source urls if the files they seem to refer to also have other urls of the same url class as the file import object being actioned (basically, this is some logic that tries to detect bad source url attribution, where multiple files on a booru (typically including alternate edits) are all source-url'd back to a single original)
  • gallery page parsing now discounts parsed 'next page' urls that are the same as the page that fetched them (some gallery end-points link themselves as the next page, wew)
  • json parsing formulae that are set to parse all 'list' items will now also parse all dictionary entries if faced with a dict instead!
  • added new stop-gap 'stop checking' logic in subscription syncing for certain low-gallery-count edge-cases
  • fixed an issue where (typically new) subscriptions were bugging out trying to figure a default stop_reason on certain page results
  • fixed an unusual listctrl delete item index-tracking error that would sometimes cause exceptions on the 'try to link url stuff together' button press and maybe some other places
  • thanks to a submission from user prkc on the discord, we now have 'import cookies.txt' buttons on the review sessions panels! if you are interested in 'manual' logins through browser-cookie-copying, please give this a go and let me know which kinds of cookies.txt do and do not work, and how your different site cookie-copy-login tests work in hydrus.
  • the mappings cache tables now have some new indices that speed up certain kinds of tag search significantly. db update will spend a minute or two generating these indices for existing users
  • advanced mode users will discover a fun new entry on the help menu
  • the hyperlinks on the media viewer hover window and a couple of other places are now a custom control that uses any custom browser launch path in options->files and trash
  • fixed an issue where certain canvas edge-case media clearing events could be caught incorrectly by the manage tags dialog and its subsidiary panels
  • think I fixed an issue where a client left with a dialog open could sometimes run into trouble later trying to show an idle time maintenance modal popup and give a 'C++ assertion IsRunning()' exception and end up locking the client's ui
  • manage parsers dialog will now autosort after an add event
  • the gug panels now normalise example urls
  • improved some misc service error handling
  • rewrote some url parsing to stop forcing '+'->' ' in our urls' query texts
  • fixed some bad error handling for matplotlib import
  • misc fixes

• version 319

  • started the new convert-query-text-to-gallery-urls object. these objects, which I was thinking of calling 'Searchers', will be called the more specific and practical 'Gallery URL Generators', or GUGs for short
  • the first version of GUGs is done, and I've written some test ui for advanced users under network->downloader definitions->manage gugs. this ui doesn't save anything yet, but lets you mess around with different values. if we don't think of anything else needed in the next week, I will fix this code for v320 and start filling in defaults
  • watchers now have a checking slot, much like the recent change to galleries and subs. it safely throttles dozens of threads so they don't rudely hammer your (or the destination server's) CPU if they all happen to want to go at once (like just after your computer wakes up). the option is similarly under options->downloading
  • moved the new gallery delay/token management code to the better-fit bandwidth manager (it was in domain manager before)
  • the gallery delay/token code now works per-domain!
  • moved the gallery delay/token checking code into the network job proper, simplifying a bunch of import-level code and making the text display now appear in the network job control. token consumption now occurs after bandwidth (it is now the last hoop to jump through, which reduces the chance of a pileup in unusual situations) I expect to soon add some kind of 'force-go' action to the cog menu
  • the network engine will now not permit more than three jobs active per domain, and the overall limit has been raised from ten to fifteen
  • the media right-click menu now supports copying: all of a files recognised urls; all of a files urls; all selected files' urls of a specific url class; and all selected files urls
  • reworked and harmonised a bunch of urlparsing and generation code--all urls should now appear as full unicode across the program, generally without %20-type encoding characters unless explicitly entered by the user. character encoding now all happens on the backend in requests
  • non-url-class-matched urls now have their query parameters alphabetised as part of the normalisation process
  • all urls in the db will have their query params alphabetised on update, and any file relationships merged to the new/existing normalised url
  • the manage urls dialog will now normalise newly added urls (but should also still permit the removal of non-normalised urls)
  • reworked how gallery hits update file import object caches, particularly for subscriptions
  • fixed an issue in subscriptions gallery logging where the gallery log would always state it had found the max number of files and typically redundantly generate an 'ignored' stub--it should now say something like 'found 7 files - saw 5 previously seen urls, so assuming we caught up' as originally intended
  • simplified some gallery->file import object creation
  • galleries now compact until 100 entries (was 25)
  • watchers now gallery-compact after a successful check
  • watchers now show the 'just added'/'already watching' status for 15s, up from 5s
  • network report mode now reports three time--once each for job addition, start, and successful completion
  • fixed an issue with the new 'max width' popup sizing calculation that was sometimes not fitting for new height requirements correctly
  • fixed an issue with the new url class next page generation code
  • fixed an issue where TIOs with data regarding since-deleted services were failing to initialise at the ui level
  • misc status text cleanup

• version 318

  • downloaders

  • extended url classes to support 'next gallery page' generation--a fallback that predicts next gallery page url if the parser cannot provide it (as is often the case with APIs and unreliable next-page-url galleries such as gelbooru)
  • integrated this new next page generation into new gallery processing pipeline
  • updated gelbooru, tumblr api and artstation gallery api url classes to support the new next gallery page business
  • fixed the url class for xbooru, which wasn't recognising gallery urls correctly
  • wrote new gallery parsers for rule34.paheal and mishimmie (which are both shimmie but have slightly different gallery layout). this should finally solve the 'one paheal gallery url is being parsed into the file list per page' problem
  • 'fixed' the tumblr parser to fetch the 1280px url (tumblr killed the raw url trick this past week)
  • misc text/status fixes
  • wrote a gallery parser for tumblr that fetches the actual tumblr post urls and hence uses the new tumblr post parser naturally! (tumblr post urls are now more neatly associated as 'known urls' on files!)
  • note that as the tumblr downloader now produces different kinds of urls, your tumblr subs will hit your periodic limits the next time they run. they will also re-download any 1280px files that are different to the previously fetched raws due to the above raw change (protip: keep your subscription periodic file limits low)
  • cut the 'periodic limit' subscription warning popup down to a much simpler statement and moved the accompanying help to a new help button on the edit sub panel
  • multi-gallery pages now have an 'added' column like multi-watchers
  • the new 'pause' ? and 'stop' ? characters shown in the multi-downloader pages are now customisable under options->downloading (some users had trouble with the unicode)
  • the watcher now shows the 'stop character' if checking is 404/DEAD
  • fixed an issue where the new gallery imports on the same multi-page were all sharing the same identifier for their ephemeral 'downloader instance' bandwidth tracker, which meant they were all sharing the same '100rqs per 5mins' etc... rules
  • the page and subscription downloader 'gallery page delay' is now program-wide (since both these things can run in mass parallel). let's see how it goes, maybe we'll move it to per-site
  • subscription queries now auto-compact on sync! this means that surplus old urls will be removed from their caches, keeping the whole object lean and quick to load/save
  • gallery logs now also compact! they will remove anything older than twice the current death velocity, but always keep the newest 25 regardless of age

  • misc

  • the top-right hover window will now always appear--previously, it would only pop up if the client had some ratings services, but this window now handles urls
  • harmonised 'known urls' view/copy menu to a single code location and added sorted url class labels to entries (which should reduce direct-file-url misclicks)
  • greatly sped up manage tags dialogs initial calculation of possible actions on a tag alteration event, particularly when the dialog holds 10k+ tags
  • greatly sped up the second half of this process, when the action choice is applied to the manage tag dialog's current media list
  • the buttons on the manage tags dialog action popup dialog will now only show a max of 25 rows on their tooltips
  • some larger->smaller selection events on large pages with many tags should be significantly faster
  • subscription popups should now 'blank' their network job controls when not working (rather than leaving them on the old job, and without flickery-ly removing the job control completely)
  • the file cache and gallery log summary controls now have ... ellipsized texts to reduce their max width
  • fixed an issue where larger 'overriding bandwidth' status wait times would sometimes show instead of legit regular smaller bandwidth wait times
  • removed a now-superfluous layer of buffering in the thumbnail grid drawing pipeline--it seems to have removed some slight lag/flicker
  • I may have fixed the issue where a handful of thumbs will sometimes remain undrawn after several fast scrolling events
  • gave the some-linux-flavours infinitely-expanding popup message problem another pass. there _should_ be an explicit reasonable max width on the thing now
  • added a 'html5lib not found!' notification to the network->downloaders menu if this library is missing (mostly for users running from source)
  • help->about now states if lz4 is present
  • gave 'running from source' help page another pass, including info on running a virtual environment
  • in file lookup scripts, the full file content now supports string transformations--if this is set to occur, the file will be sent as an addition POST parameter and the content-type set to 'application/x-www-form-urlencoded'. this is a temp fix to see if we can get whatanime.ga working, and may see some more work
  • if the free space on the db dir partition is < 500MB, the program will not boot
  • if the free space on the db dir partition is < 1GB, the client will not sync repositories
  • on boot the client can now attempt to auto-heal a missing local_hashes table. it will give an appropriate error message
  • misc post-importing-cleanup refactoring

• version 317

  • completely overhauled the tag filter panel

  • the tag filter panel now has 'whitelist' and 'blacklist' pages beside the old 'advanced' sub-panel. these new simple pages are much more human friendly for common workflows and provide easy-select checkboxes for namespace classes (which are compiled from all the namespaces your parsers can currently do)
  • the tag filter rule entering workflow now stops you from creating overcomplicated rulesets: when adding a blacklist rule, it will now only add an explicit entry if it is not already blocked by a higher rule (otherwise it will just discard from whitelist, if there)--and when adding a whitelist rule, it will now only add an explicit entry if it is already blocked by a higher blacklist rule (otherwise it will just discard from blacklist, if there)
  • tag filters now provide more human-friendly summary statements
  • misc improvements to tag filter ui logic
  • the various help texts surrounding the tag filter panel all got passes
  • the tag filter panel now uses text-and-paste controls for mass-adding of tags
  • namespace checkboxes have been completely removed from the tag import options panel and various other related places. any existing TIO with checked namespaces will be automatically updated to 'get tags' with an appropriate filter. this is an important step in the rewrite--everything is now handled in the new tag filter panel
  • simplified and sped up the actual tag filtering code

  • numerous multi-importer improvements

  • the gallery and watcher page lists will now ~dynamically~ resize in height based on number of entries, from roughly four columns to twenty four. this relayout code somehow seems to work on all platforms
  • sped up the 'results loading' step of gallery/watcher highlighting immensely--on a typical list of a couple hundred files, it should now be about 50ms total (before, depending on presentation rules, it could be 0.8-3s)
  • added an additional db-skipping optimisation for calculating presentation status
  • watcher and gallery highlights will now filter out trash and completely deleted files (the ones that appear with a dark default 'hydrus' icon) on reloads
  • added two checkboxes to options->downloading for 'if nothing is highlighted when I add a new X, highlight that new X' for watchers and galleries
  • adding or removing a query or watcher from the new multi-lists should now be reflected in the list ui instantly, rather than after a <=1s delay
  • added url classes and parsers for imgur single and multiple urls--thanks to the community for providing some examples
  • added url class and parser for derpibooru single file pages--again thanks to the community. derpibooru hence now supports basic drag and drop import
  • fixed an issue where the watcher was often still checking despite 404 status
  • watchers and galleries use a little less CPU to update some of their ui
  • added simple subsidiary page parsing support to file import objects (previously, this only worked in the gallery log)

  • thumbnails

  • gave the thumbnail scrolling code a pass--it is now a bit cleverer about drawing and uses a larger number of smaller 'tile' bmps rather than pages
  • added an 'EXPERIMENTAL' option to options->gui to change the number of thumbnails each scroll tick scrolls. it defaults to 1.0, but you _should_ be able to set 0.5, 0.37, whatever. please report any bugs!
  • added a thumbnail debug mode to help see the new thumbnail layout boundaries

  • misc

  • the max subscription file limits are now 10,000 for users in advanced mode
  • the default subs initial/periodic limit is now 100/100 (bumped up from 100/50)
  • the file import dialog now has a little cog icon to change whether human sort is applied on path addition events (for e.g. if you want to add in some date order from an explorer window)
  • humansort now sorts case-insensitive
  • by default, unmatched urls will no longer display in the top-right of the media viewer. see how you like this and let me know if you would like an option to put them back
  • the speed text on the right-side of the network job control now dynamically resizes to its min size, which gives the text on the left side (where it is often cut off, saying 'overriding bandwidth ...') more space when available
  • I think I fixed an issue where the popup frame could spam-resize in odd ways (such as growing a pixel wider every update tick)
  • watchers will no longer include the '* ' highlight prefix in subject-based sort comparisons
  • in prep for an eventual major code refactoring, the thumbnails' underlying media object now stores a faster db-based numeric file identifier
  • 'duplicate' calls on the new listctrl will now insert the dupes in the current correct sort location, rather than tacking them on the end
  • drag and drop imports to the new listctrl will also now insert like this
  • caught up edit subscriptions panel to the finalised common listctrl panel code, including the import/export/duplicate buttons
  • the multiple checkboxlist selection dialog now sorts by label
  • converted all old checkboxlist dialogs to the new panel system
  • massively sped up certain kinds of parsing that were wasting time hitting a cache test way too often
  • fixed an old hash filtering system
  • moved to a simpler and more stable way of calculating certain text extents
  • fixed an issue where the include directory (which has the original source, which isn't a big deal but is nice to have) wasn't being correctly copied into the linux build
  • the os x .tar.gz build now has the include directory
  • refactored some client tags code around
  • misc cleanup

• version 316

  • gallery

  • gallery url classes can now be linked to parsers!
  • if parsers are linked, gallery pagewalk can now work on the new parsing system. gallery import pipeline has been significantly updated to reflect this
  • gallery import objects are now 'multiple' gallery imports, much like the multi-watcher, with each separate query having its own entry in a list (they also run in parallel!)
  • the multi-gallery list will show file/gallery pause status in slender columns, and will show a 'stop' character when gallery parsing is done
  • wrote a 'gallery selector' button and added it to the new multi-gallery page, so you can spawn queries for ~different sites~ on the same import page! it always defaults to 'deviant art' for now, but when the next 'searcher' overhaul step is done, this will be customisable
  • the new page selector and related 'pages' menu is now simpler--with the new selector, you just select 'gallery'
  • added 'new_gallery_downloader_page' shortcut action to the 'main gui' set to allow quick opening of this new page type
  • wrote a 'gallery import panel', which reviews a single gallery import stream, and added it to the multi-gallery page to show the current highlighted query
  • as all gallery imports now run in parallel and work on the new system, the now almost-useless 'cancel' gallery pagewalk button is now removed
  • with the wider availability of the new gallery log for file count and error reports, shifted around and smoothed out some gallery status text presentation
  • improved the auto url_class->parser linking 'try to fill in gaps' logic to work with gallery urls (this was surprisingly complicated)
  • fixed a misc stupid waste of time in auto url_class->parser linking
  • many misc updates to gallery pipeline

  • subscriptions

  • wrote a new gallery pagewalk pipeline for subscriptions, which still does oldest-to-newest url addition and obeys file limits and so on
  • numerous subscription pipeline and error handling tweaks and improvements, particularly in regards to the new code
  • subscriptions now have max initial and periodic file limits of 1000. existing subs with >1000 or infinite will be cut to 1000. there is a help button on the edit sub panel to explain why you should do large syncs with the manual downloader and not subs
  • the ugly and dangerous-if-you-scroll-in-the-wrong-place gallery selector mismash control in the edit subscription panel is now replaced with the new gallery selector button
  • fixed an issue where the edit subs panels could sometimes say '48 years ago' (i.e. displaying a literal time delta since 0, 1970) on initial timestamps
  • juggled some 'periodic limit' reporting logic to skip an unusual false positive that affect hentai foundry subs for now and more in future

  • urls

  • the url downloader now accepts gallery urls and will receive drag-and-dropped gallery urls. at the moment, it only parses the one page (i.e. it doesn't start a new 'searching' pagewalk) and sends the parsed links to its file queue

  • watcher

  • finished 'watcher panel', which reviews a watcher, and added it to the multi-watcher page to show the current highlighted watcher
  • the single watcher page is completely removed--it is only the multiple watcher now. all singles will be converted to multiples on update
  • some single-watcher options (like watchers naming their own page tabs and the [404]-style page name prefixes) are removed
  • multiple watcher panel now lists file/checking pause status and has separate buttons to control these paused statuses
  • fixed some misc watcher highlight code--highlighted watchers should correctly publish to the page from the start of session load now
  • improved some 'repage' logic in how highlighted threads get removed

  • misc

  • discovered a scroll-setup parameter that stops janky scroll-to-click-focus behaviour on all the new scrolling panels, thank the LORD
  • improved some 'can't parse' error handling for post url parsing
  • reworked how all importers present their network jobs to the ui, including fast response if the switch happens during a job
  • in prep for searcher switchover where all downloader sources will be harmonised into one system, booru identifiers now present in several ui locations as 'name', not 'booru: name'
  • updated the danbooru parser to deal with the new 'next page' markup they use
  • wrote a gelbooru gallery parser that works with 0.2.0 and 0.2.5 gelb sites--an ancillary issue where gelb-related downloaders could sometimes not accurately figure out the magic '42' next-page offset is hence now fixed
  • wrote an e621 gallery page parser
  • these sites hence now support single-page gallery drag-and-drop
  • added url classes for artstation gallery url and its api counterpart, but didn't go further yet--we aren't quite there with api pagewalking just yet
  • updated deviant art gallery url classes
  • added an e621 'search initialisation' gallery url class to improve some future drag-and-drop stuff
  • url normalisation no longer cuts off 'www.'-style prefixes
  • url comparison is more careful to test 'www.'-style prefixes, so a file import cache should recognise that 'http://www.blah.com/blah' is the same as 'https://blah.com/blah'
  • did a bunch of refactoring to further split up the bloated ClientImporting.py
  • fixed some misc downloader layout that may have been hiding some texts previously
  • some multi-watcher and multi-gallery events like add/pause query should be a bit snappier
  • in the parsing ui, url and title priorities are now 50 by default
  • prepped a little subscription unit test code for when searcher object is done
  • misc downloader layout improvements
  • misc listctrl refactoring

• version 315

  • got started on the big gallery update, but decided not to pull the trigger just yet. I hope to do it next week, switching the whole thing over to a two-object multi-watcher kind of deal
  • updated to wxPython 4.0.3 for all platforms
  • cleaned up some menubar replacement code, and the update to the new wxPython should also fix a "event for a menu without associated window" bug some gtk2 users were seeing on quick menubar changes
  • manage default tag import options panel now has copy/paste buttons that work on the listctrl
  • added some 'paste tag import options' safety code to make sure no one accidentally pastes a subscription or something in there, wew
  • added default checker options for subscriptions to options->downloading
  • unified how checker options are edited from their button, much like how file and tag import options work. it also has a summary tooltip on the button
  • the checker options under options->downloading are now these slimmer buttons
  • in the manual import dialog (which pops up when you drop a folder/files on the client), the files will now be added in 'human friendly' number sorting, so files of the sort 'Favourites - 10.jpg' will sort [10, 11, ..., 99, 100] rather than the purely lexicographic [10, 100, 11, ..., 99]
  • gave the migrate database dialog a pass--a bunch of misc presentation changes and a general simplification of workflow, now based more on just increase/decrease location weight
  • a bunch of texts on page management (left-hand) panels that share horizontal space with buttons should now ellipsize ("downlo...") when they get too long for the width instead of drawing in an ugly way over the buttonspace
  • moved the manage import folders dialog to the new listctrl and added a 'paused' and better 'check period' column
  • if a user tries to run a 'paused' import folder specifically from the menu, the import folder will now unpause (I will probably remove this old paused variable in the future--it isn't of much use any more)
  • tightened up some repository reset code that wasn't deleting all service tables and hence recovering from some service id malformation errors correctly
  • wrote a 'clear orphan tables' db maintenance routine that kills some spare tables some users who have previously deleted/reset repositories may have floating around
  • fixed an issue with parsing folders after hitting cancel button on the import files pre-dialog
  • if watchers encounter non-404 network errors during check, they should now just delay checking for four hours (before, they were also pausing checking completely)
  • if watchers are in 'delay' mode, they'll also not work on files.
  • file and gallery downloads that hit a 403 (Forbidden) will now present a simpler error status, like they do for 404
  • the new post downloader will no longer fail if one of the parsed source urls is not a url. the borked string will also not be associated as a url
  • regular gallery downloads now override bandwidth for the file download step, which is almost always the second half of a pair of post_url/file downloads, just to keep things in sync in edge cases
  • cleaned up some timestamp generation and 'overriding in x seconds' strings to be more human friendly
  • improved some serverside file parse error handling to propagate the actual error description up to the client a bit better
  • fixed typo causing incorrect num_ignored count in file import status button right-click menu
  • parseexceptions will now present more data about which page and content parser caused the problem. I am not totally happy about how this solution works and may revisit it
  • the lz4 import error catching is now more broad to catch some odd problem I discovered in new Linux build environment
  • the moebooru parser now fetches the original png of an image, if available
  • added a new tumblr parser that also gets post tags--it _shouldn't_ be the default
  • the new login pipeline now kicks in for the legacy logins--pixiv and hentai foundry--on a per-url basis, so adding pixiv/hf urls to the url downloader will trigger a login even if needed (previously, this was tied to legacy gallery initialisation, which explains some pixiv 'missing' login stuff some users and I were having trouble with)
  • if the legacy login system fails in the new pipeline, it now sets a flag and won't try again that client boot
  • the old 'default tag import options' panel is now completely removed from options->importing. please check 'network->downloaders->manage default tag import options' for the new url-based settings
  • misc fixes

• version 314

  • tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway
  • updated tag import options ui to enable this default setting where appropriate
  • updated the newer import pipeline to work with 'default'-set tag import options
  • new downloaders, subscriptions, watchers, and multi-watchers now start with 'default' tag import options
  • deleted the old default tag import options management code and put some text up on options->importing making notice about the impending shift. a popup message will also say this on update
  • tag import options buttons now have a right-click menu with copy/paste/default options for quick assignment and duplication!
  • added 'overwrite tag import options' to manage subscriptions--once you are comfortable with the new 'default' TIO mode, and after some small tests, you might want to switch all your subs over to 'default'
  • the 'urls' downloader now has a tag import options--it initialises as 'default'
  • added furry.booru.org to gelbooru 0.2.0 parser
  • wrote a hentai foundry file page parser
  • wrote a moebooru file page parser (this works for konachan, yande.re, and sakugabooru)
  • wrote a shimmie parser (this works for rule34.paheal, rule34hentai, and mishimmie)
  • wrote a newgrounds parser
  • integrated the user-created sankaku parser
  • wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast
  • (hence all these above sites now support drag and drop!)
  • rolled out some new tumblr url classes to handle all this.
  • added sakugabooru url classes
  • fixed an issue where url classes were not normalising api urls in all cases, meaning some url classes would not api-link correctly in 'manage url class links' panel
  • fixed an issue with deviant art legacy gallery parser pulling some funky 'creator:' tags
  • some misc new downloader error handling improvements
  • the watcher now uses the new gallery object to parse and generate file import objects
  • the downloaders with gallery logs should now report non-success gallery fetches, along with error tracebacks (this will include some no-worry 404s the legacy downloader sometimes uses to terminate searches)
  • added image and thumbnail cache timeout time delta buttons to options->speed and memory
  • added a 'show the D on short file import summaries' checkbox to options->downloading--it defaults to off
  • the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion
  • added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated
  • added a checkbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on
  • fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add
  • shook up and collapsed the network menu into neater categories
  • tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text
  • if database errors include the word 'malformed', the client now throws a little extra error text pointing people to the help.txt in the db dir
  • cleared out some legacy download code
  • cleared out legacy hard drive import error handling, moving it all to the new file import object
  • misc refactoring and cleanup

• version 313

  • fleshed out the new gallery log and its constituent log entry objects
  • added gallery logs to gallery downloaders, subscriptions, url downloaders, simple downloaders and watchers
  • added very simple gallery log reporting to these downloaders
  • added first, read-only version of gallery log ui to these downloaders
  • fleshed out some new gallery/file-object pipeline stuff
  • wrote a simple danbooru gallery page parser and added it on update. it doesn't do anything yet, but if you are into the new parsing system, please check it out as an example
  • the url downloader now has a full file import status control with status text
  • fixed a url count issue on completely fresh gallery downloads that was stopping gallery searches one file (like 199 vs 200) before the file limit
  • the pixiv downloader now fetches 'type=all' gallery pages, which include specifically manga file pages (as opposed to merely multi-file 'illustrations')
  • added 'retry ignored' to the file import status button's right-click menu
  • fixed the deviant art url class and parser to use the new file page format. also added an '(old format)' class to match the old way for legacy purposes (this legacy class also uses an api conversion to connect to the new parser--we'll also figure out a way to convert all these over at the db level en masse later!)
  • updated some similar deviant art gallery stuff as well
  • tag import options now has a tag filter to go along with the 'get all tags' checkbox! ('get all tags' is now renamed to 'get all' as a result). this filter lets you make more complicated tag filtering decisions like 'get all tags except "species:" tags'.
  • the new 'only get tags if they already exist' checkbox now also has a filter, if you want to only apply this test to a subset of tags (like the unwashed mess of unnamespaced tags many boorus and sites provide)
  • generalised a 'tag filter' button class to make it simpler to edit tag filters across the program, and cleaned up some related status code
  • fixed a problem with deriving tag import options for specific url classes when that url class was part of an api-url-class chain
  • if the domain manager cannot now find a url match for a pending download, it now assigns the file post default tag import options to that import
  • added a new 'duplicates' options page that has a hacky way to edit the weighted scores used to determine which of the pair of files to present file in the duplicates filter
  • unifed how some file import status generation works, adding a new 'simple status' string to briefly summarise progress in multi-watcher and edit subscriptions columns
  • cleared out some old redundant status caching in the urls downloader
  • simplified how almost all timestamp strings are generated
  • simplified how time delta strings are generated
  • brushed up some simple common ways to present timestamps as 'human pretty' strings
  • all places where timestamps would be presented as a mix of '5 days ago' and complete datetime strings will now present as '5 days ago' unless you set the new options->gui 'always show iso' checkbox. going back to simple to clear up confusion in workflow and code. I may revisit this, as turning on ISO mode now spams it all over the place
  • cleaned up the 'looks like the computer just woke from sleep' check and reduced its grace period to fifteen seconds. foreground daemons (like the subscription daemon) and the network engine will now also obey it
  • added a 'simulate wake from sleep' debug action to better test the sleep-wake detection code
  • improved my custom statictext class to auto-wrap text without flickering
  • used this new autowrapping to improve wrapping and layout of popup message texts
  • replaced all other st wrapping with this new code
  • wrote a little helper function to better dedupe lists in future
  • did a bunch of refactoring to neaten some long common func names
  • deleted some old unused code

• version 312

  • converted much of the increasingly complicated tag import options to a new sub-object that simplifies a lot of code and makes things easier to serialise and update in future
  • tag import options now allows you to set whether tags should be applied to new files/already in inbox/already in archive, much like the file import options' 'presentation' checkboxes
  • tag import options now allows you to set whether tags should be filtered to only those that already have a non-zero current count on that tag service (i.e. only tags that 'already exist')
  • tag import options now has two 'fetch if already in db' checkboxes--for url and hash matches separately (the hash stuff is advanced, but this new distinction will be of increasing use in the future)
  • tag import options now applies sibling and parent collapse/expansion before tag filtering, which will improve filtering accuracy (so if you only want creator tags, and a sibling would convert an unnamespaced tag up to a creator, you will now get it)
  • the old 'all namespaces' checkbox is now removed from some 'defaults' areas, and any default tag import options that had it checked will instead get 'get all' checked as they update
  • caught up the ui and importer code to deal with these tag import option changes
  • improved how some 'should download metadata/file' pre-import checking works
  • moved all complicated 'let's derive some specific tag import options from these defaults' code to the tag import options object itself
  • wrote some decent unit tests for tag import options
  • wrote a parser for deviant art. it has source time now, and falls back to the embedded image if the artist has disabled high-res downloading. if it finds a mature content click-through (due to not being logged in), it will now veto and set 'ignored' status (we will revisit this and get high quality nsfw from DA when the login manager works.)
  • if a check timings object (like for a subscription or watcher) has a 'static' check interval, it will now apply that period to the 'last next check time', so if you set it to check every seven days, starting on Wednesday night, it will now repeatedly check on Wed night, not creep forward a few minutes/hours every time due to applying time to the 'last check completed time'. if you were hit by this, hit 'check now' to reset your next check time to now
  • the multiple watcher now sorts by status by default, and blank status now sorts below DEAD and the others, so you should get a neat subject-alphabetical sort grouped by interesting-status-first now right from the start
  • added 'clear all multiwatcher highlights' to 'pages' menu
  • fixed a typo bug in the new multiple watcher options-setting buttons
  • added 'retry ignored' buttons to edit subscription/subscriptions panels, so you can retry pixiv manga pages en masse
  • added 'always show iso time' checkbox to options->gui, which will stop replacing some recent timestamps with '5 minutes ago'
  • fixed an index-selection issue with compound formulae in the new parsing system
  • fixed a file progress count status error in subscriptions that was reducing progress rather than increasing range when the post urls created new urls
  • improved error handling when a file import object's index can't be figured out in the file import list
  • to clear up confusion, the crash recovery dialog now puts the name of the default session it would like to try loading on its ok button
  • the new listctrl class will now always sort strings in a case-insensitive way
  • wrote a simple 'fetch a url' debug routine for the help->debug menu that will help better diagnose various parse and login issues in future
  • fixed an issue where the autocomplete dropdown float window could sometimes get stuck in 'show float' mode when it spawned a new window while having focus (usually due to activating/right-clicking a tag in the list and hitting 'show in new page'). any other instances of the dropdown getting stuck on should now also be fixable/fixed with a simple page change
  • improved how some checkbox menu data is handled
  • started work on a gallery log, which will record and action gallery urls in the new system much like the file import status area
  • significant refactoring of file import objects--there are now 'file seeds' and 'gallery seeds'
  • added an interesting new 'alterate' duplicate example to duplicates help
  • brushed off and added some more examples to duplicates help, thanks to users for the contributions
  • misc refactoring

• version 311

  • wrote a new parser that muddles its way through pixiv's new dynamic javascript layout. it seems to get everything working again. it gets tags in kanji, although the unnamespaced pixiv tags remain low quality, and you may wish to just not parse them at all anyway
  • fixed some misc parser text handling, unicode conversion etc...
  • the new pixiv parser has a 'page' tag stub that should inform tag import options in the old downloader
  • the multiple watcher now remembers the highlighted watcher through a session restart
  • the multiple watcher now shows the highlighted watcher's url up top
  • the multiple watcher now has checker, file import, and tag import options, which it will assign to all new watchers it creates
  • the multiple watcher now has a 'set options to watchers' button that will force-set the current options to all the selected watchers
  • the multiple watcher now has an 'added' column with watcher creation time listed. storing this creation time is new, so any existing watchers will get a new creation time of their next load time, but it is remembered henceforth. the listctrl here is now pretty crushed for width, so maybe we'll rejigger some stuff here
  • watchers added to a multiple watcher will now have a status of 'just added' for five seconds
  • watchers that are added to a multiple watcher that is already watching them will now have the status of 'already watching' for five seconds
  • the multiple watcher list now has a much taller minimum height--layout here is another work in progress
  • fixed the inkbunny parser (and a related tweak to the inkbunny url class)--it now uses the new 'multiple-file-per-post' import object generation to actually walk through the pages of the mini-gallery (which for inkbunny have -p2- suffixes on the url) to fetch only the correct files and url-associate them neatly
  • tag import options now has a 'get all tags' checkbox, which can override the normal namespace checkboxes. it gets all tags, even those with namespaces not listed, which happens for several reasons in the new download system. (eventually, the namespace list may be replaced with a slightly different system)
  • watcher tag import options no longer list 'filename' under their namespace checkboxes--they just have this 'get all tags', which works for everything (so watching yiff.party pages should now get tags)
  • simplified and sped up similar files search at the db level
  • sped up some ratings search code

  • generalised some common file search optimisations, meaning they now apply in more situations and can take advantage of some other speed-ups

  • similar files system predicate is now faster
  • inclusive ratings searches are now faster
  • duplicate relationship count searches with non-zero-inclusive count are now faster
  • removed some clumsy old ratings search optimisation code
  • exporting serialised objects as pngs is a bit easier--now, it displays current export path better, will remember the last export location used, and for single png exports will pre-fill the filename and 'title' value with a reasonable default
  • the content parser, page parser, and url class listctrls now accept serialised png files when drag and dropped!
  • the simple downloader should recover and continue better from malformed urls during a page parse
  • the url downloader should now recover better from various situations where it cannot not derive some tag import options (including urls with a 'file' url class, such as 4ch/8ch direct file links)
  • parse test results will now state the priority value of urls
  • gave the 'updating' section of help a pass and wrote a little more on how to do a big-version-gap update
  • when a new multi-file import object inserts its child file import objects while being looked at in the ui, the listctrl should now correctly refresh the displayed indices
  • subscriptions will now wait up to 90s for bandwidth (was 30s before, I think) before quitting, which should avoid a few more early-quit events
  • cleaned up some server decompression bomb testing
  • users with admin-level accounts can now upload decompression bombs to file repositories, better options on this will be avaliable in future
  • the manage urls dialog will now OK on the same 'manage_file_urls' shortcut action that can open it (like manage tags and ratings already do)
  • fixed the string converter for new file lookup parsing scripts
  • started work on some in-the-background mass file reparsing, but I want to get some nicer ui going before I pull the trigger on any of it
  • file reparsing now repopulates the table for md5, sha1, and sha512 hashes if they are missing
  • improved some ffmpeg error parsing
  • moved from basic list to a pop-faster collections.deque for importable path parsing and duplicate search branch regen
  • added a BUGFIX option to options->gui that forces minimum width for popup messages in the continuing attempt to deal with some funny fit/layout calculation in certain Linux WMs
  • fixed how some 'unrepairable db' error messages are displayed in Linux systems
  • cleaned up a ton of old tuple-stripping code from the db
  • updated to new sqlite for windows build
  • misc improvements

• version 310

  • updated the inkbunny file page url class to acknowledge that inbunny pages can have multiple files
  • updated the inkbunny file page parser to handle multiple file urls (although they may be out of order and possibly sometimes include the artist profile image--this was not super easy)
  • added a parser for twitter tweets (only images supported atm, but it can handle multiple!) (hence tweet drag and drop now works!)
  • updated the artstation file page url class to redirect to a new api url class
  • wrote an artstation file page parser that also handles multiple file urls
  • updated/added pixiv file page, manga page, and mange_big url classes
  • updated pixiv file page parsers to be ok with manga links
  • wrote parsers for pixiv manga and manga_big pages to fetch manga files (with page tags)!
  • file import objects can now create semi-duplicate children for multi-file post urls and insert them just after themselves in the file import queue.
  • file import objects can now receive and remember referral urls. this referral url is associated with the file if appropriate. the watcher and simple downloader now uses this in addition to the multi-file post system
  • jumbled around some parameters and merged the two new file import url commands (import 'file' vs import 'post') into one single simple 'work on this url, thanks' call that is now used across the program
  • the parsing system's 'content parser' no longer fetches file urls and post urls, but 'download urls' and 'source urls'. this helps some pipeline logic and also lets post urls be download urls
  • when file import objects parse post urls as the urls to download, it now creates 1-n new import objects, just like if multiple file urls.
  • improved some file import object file association code
  • the new parsing system will de-dupe parsed urls
  • refactored the 'seed' code, which handles all basic file import objects, to the new ClientImportSeeds.py
  • added a new string transformation type, 'integer addition', for shifting page number tags up and down
  • fixed thumbnail generation for some videos that failed to do the new x%-in generation--it reverts more reliably just to the old frame 0 method
  • file reparsing popup now has a stop button
  • fixed an issue where extremely thin or wide (ratio > 200:1) images would not generate a full-size or resized thumbnail
  • the file reparsing/re-thumbnailing now reports errors better (including with full path) and does not abandon the larger job as it works
  • misc thumbnail generation code improvements
  • improved some thumbnail and file regeneration/moving code when the existing file has read-only status
  • the multiple watcher now has a 'check now' button
  • added a checkbox to options->gui that will put new notebook page tabs on the left
  • for all file download network jobs working in the new download system, the file import options for min size, max size, and max size (gifs) are now applied _during the download_! if the server tells the client the exact file size in the response headers, it will test max and min size before the content is actually downloaded--otherwise, it will test the max size as it downloads. if the server clearly says the file is a gif, the max gif size rules will also be tested in the same way
  • cleaned up some bandwidth announcement code--now, if bandwidth is due in less time than override time, that will now correctly be the status text
  • the bandwidth status no longer says 'in in' typo
  • fixed up some tag repair code from last week
  • the 'print garbage' debug function now dumps a whole bunch more data to the log
  • the thumbnail cache should now be a bit more stoic about missing repository thumbnails--it should now just present the hydrus default backup without error popup spam
  • the repository thumbnail sync will now get as thumbs in blocks as high as 10k at a time, rather than the old 100
  • hydrus network requests no longer generate web domain network contexts (and so won't have a default one-request-per-second bandwidth limit and should stream through thumbnails a bit faster)
  • hydrus network services are now willing to wait longer for bandwidth, so big thumbnail queues should keep working even if other bottlenecks pause them for a bit
  • hydrus network services will no longer sometimes have double-sync popups if synced from the advanced 'sync now' button in review services
  • changed the default global 'stop-accidents' bandwidth rule of 120rqs per minute to 512MB per minute. this only affects new users, but users trying to sync to large file repos might like to make a similar change manually
  • doing giant full file delete (i.e. purge from trash) jobs should now be a bit gentler on the gui
  • improved how the client deletes paths, clarifying in the code when and when not to allow recycle (usually disabled for thumb disposal)
  • switched the hacky text widgets on the popup system to a newer object. seems to still render ok, so lets see if it fixes some unusual layout issues some users have seen
  • if the temp folder cannot be created on boot, the client will continue anyway
  • fixed some url-domain text handling in db storage that was also breaking v309 update for some users
  • fixed some additional domain generation error handling at the db level
  • the list of url classes in the system:url panel is now the list of all url classes that are considered associable (before, it was file and post urls)
  • if a url class now api-links to itself or otherwise forms a loop with n other api url classes, the client will now throw an error (rather than lock up in an infinite loop!)
  • in the parsing ui, tag parse test results are now cleaned before being displayed
  • fixed misc url matching error reporting bug
  • when consulting the current file limit, the gallery page downloader will now try, when it has that number, to consult the total number of urls found it the current search (old behaviour is to only consult the number of _new_ urls, which lead to some bad edge-case workflows)
  • misc refactoring

• version 309

  • wrote a fix for the tumblr GDPR issue under _network->DEBUG: misc->do tumblr GDPR click-through_. you will also get a popup about this on update
  • the tumblr downloader will try to detect the GDPR problem and present a similar popup guiding you to the GDPR click-through solution
  • the client and server now generate video (but not gif yet) thumbnails 35% in by default. the client can now change this percentage value under options->media. this was highly requested and was being put off for a longer rewrite, but I figured out a simple way to hack it in. please let me know if you get failures
  • on adding a parent, all files with the child tag will now also get all applicable grandparents (with no limit on recursive generations and dealing with accidental loops)
  • on adding a sibling, all files with any of the siblings will now also get all applicable parents and grandparents for the whole group. a maintenance call to retroactively fill in the sibling/parent gaps that are now filled will also come soon
  • this logic still does not apply in cross-service situations, which _will_ likely have to wait for a big data/gui overhaul and us figuring out what we actually want here
  • added a simple pause/play button to the multiple watcher
  • if the multiple watcher is set to catch watchable url drag and drop events and the current page is a multiple watcher, this current page will catch those new urls (as opposed to the _leftmost_ multiple watcher)
  • improved some thread unpause logic which was failing to lock pause during 404 status
  • the multiple watcher should now ignore case when it sorts by subject
  • added url class and file page parser for inkbunny (so this site is now supported in drag and drop!). it fetches creator tag, some artist-made unnamespaced tags, source time, and md5
  • added file page parser for gelbooru 0.2.0, which by default works for rule34.xxx, tbib, xbooru but certainly should work for a bunch of others. it fetches source time and source url
  • html formula parsing rules can now additionally test the tag 'string' using a standard StringMatch object. this greatly helps to parse otherwise indistinguishable 'a' tags that have string 'Original image' and so on

  • the 'have I seen this url's file before?' pre-import test is now much more strict and will cause fewer accidental false-positive 'already in db'/'deleted' results

  • the url pre-import test now does not trust source urls if they do not have a url class
  • the url pre-import test now no longer trusts urls that are supposed to only be mapped to one file but are actually mapped to multiple
  • this url pre-import test now treats url-classless original post urls and intended file urls with a special level of trust
  • urls are now stored in the db in a more powerful and in-future easily searchable way--your db will take a moment to convert to the new format on update
  • did some prep work for multi-file post urls (like pixiv manga) but did not have time to finish it
  • the filename tagging options panel (in the 'add tags based on filename' of file import dialog and import folder dialog) now updates its tags/list 0.5s after the last change event, which means typing on a giant list will not cause megalag
  • improved stability of some client-screen coordinate conversion
  • misc bmp handling stability improvements
  • improved some parsing ui stability when example data gets set after the dialog is closed
  • improved some misc dialog close stability
  • converted all but one final ui update timer to the new job scheduling system
  • there are still problems with linux stability--I will continue to work on it
  • an ugly (but basically harmless) shutdown exception sometimes caused by Animations being a bit slow on deleting their underlying bmps _should_ be fixed
  • the export files dialog now generates its paths in sort order, meaning (1), (2) de-dupe filename suffixes should now be generated nicely in order
  • the network domain manager should now always chase API URL links to get the right parser
  • made some 'the db is broke, let's try to fix it' tag recovery code more forgiving
  • misc improvements to some media indexing backend, which may fix some unusual session ghost files
  • fixed the 'sure it is ok to close this importing page' dialog to also veto on a 'cancel' event, rather than just a 'no'
  • added a guide to database_migration.html on how to move the db from just an HDD to straddle both an SSD and HDD.
  • cleaned up the help->debug menu a bunch
  • added run fast/slow memory maintenance calls to help->debug->data actions
  • misc cleanup

• version 308

  • the multiple watcher will now discard new urls if it is already watching them
  • the multiple watcher will list x/y progress as just 'x' if x==y (making it easier to scan the list)
  • the multiple watcher now lists a couple of 'total' summary lines on its ui--the top lists total number of watchers and queue progress, the bottom lists the usual '23 successful, 3 deleted' line, but summed for all watchers
  • the multiple watcher will now warn you if you try to remove the highlit, alive or un-caught-up watchers
  • the multiple watcher will now resort if the thread subject (or rather, any data in the current sort column) changes (which usually happens right after it is added, when you see it change from from 'unknown subject' to 'mlp is kino')
  • fixed an issue where multiple watchers were not unscheduling down their update job correctly on page close
  • the booru selector in the edit subscription panel should now be in the tab traversal order for keyboard/automated focusing tasks
  • the boorus in that selector are now alphabetised
  • tag import options namespaces are now alphabetised
  • removed/renamed pretty much all references to 'thread' in the watcher code and ui presentation, since it can now do a bunch of other stuff. it is now just the 'watcher' and the 'multiple watcher'
  • deleted a bunch of old static thread watcher and page of images code from the old downloading system
  • added an experimental 'compact' button to advanced mode users' manage subscriptions panels. this removes urls from the selected subscriptions' caches that are no longer useful, keeping their load/save snappy. this is still in testing--be careful with it!
  • the hydrus splash screen now has a bare frame caption and will appear in the taskbar--which helps with some alt-tab and 'where the hell did it go?' stuff if you need to enter a password
  • wrote five 'reasonable defaults' buttons for the 'check timings' options panel for quick entry for different thread/subscription scenarios.
  • added a checkbox to this panel that will swap the reactive options with a simpler single checkbox
  • also clarified/fleshed out the help button on this panel
  • fixed an important source of program instability related to page alive/dead status checking that was inadvertantly talking subtly to the main gui frame even on non ui threads
  • improved how some 'page is closed but not destroyed' test logic for pages inside a closed-but-not-destroyed notebook
  • fixed another small place where the db was talking to the main gui object about status bar updates in a potentially unstable way
  • fixed another small place where the foreground daemons were talking to the main gui frame in a trivial but potentially unstable way
  • played around with some taglist sizer and layout settings
  • the gallery and simple download pages are now a little shorter--the pause and cancel buttons are now just to the right of the status texts, rather than on their own row beneath the network job controls.
  • the various bandwidth-overriding network jobs in the download system--like gallery page downloading--now wait 30s before overriding their bandwidth. hence these jobs will now obey the usual bandwidth rules up to a point
  • the simple downloader also obeys the usual bandwidth rules for 30s but no longer has a static wait, so it can run much faster in certain situations
  • network jobs that will override bandwidth in the future will now report that countdown in their status texts
  • fixed a bug in the old booru code that meant some boorus were superfluously requesting the 0th indexed page of a gallery more frequently than needed in order to reestablish a 'page size' cache. this value is now cached globally and will be replaced by a completely different system in the new gallery downloader
  • added a decent tooltip to the 'gallery fixed delay' widgets in the options->downloading panel
  • the autocomplete input should clear itself after a 'broadcast' event a bit quicker and stop some dupe inputs in certain edge cases
  • the tumblr url class now recognises that tumblr posts can have multiple files, which helps some source url lookup logic
  • added a url class for artstation file pages
  • the primary file import url (the one listed in the file import list) will now correctly not associate with the resulting file if its url class is so set
  • all the import objects now have much lower idle CPU time and thread needs and start in slightly offset times, smoothing out the thread count spikes
  • all the import objects will now respond quickly to changes to the underlying file import cache (like right-click->try again events)
  • the new job scheduling system now uses two queues--fast and slow, in order to reduce some resort/insert overhead
  • a couple more improvements to the new job scheduling system to smooth out spikes
  • if the temporary path override does not exist, the client will now compain with spammy popup messages and fall back to the default
  • if the temporary path override does not exist or is not writeable-to on options dialog ok, a veto exception will be raised
  • refactored the watcher and multiple watcher to their own file, ClientImportWatchers
  • misc fixes

• version 307

  • wrote a gelbooru 0.2.5 (which matches gelbooru itself) parser in the new system. it now has some more redundancy and produces md5 hash and source urls
  • fixed the e621 parser for flash files
  • manage tags is now a notebook rather than a listbook!
  • a problem where OS X was displaying the wrong label in the manage tags listbook is now fixed (as it no longer uses the buggy old listbook)
  • improved autocomplete focus setting in OS X in general
  • moved all paged importer loops to the new job scheduling system, which will massively cut down on idle time thread count (and some idle CPU usage) on clients with a bunch of import pages open
  • the main thread pool (which the job scheduling system uses) can now temporarily grow much larger (200 threads) when it gets hammered
  • recalibrated some thread watcher code to deal with being embedded in a larger object better (and sharing a page with other watcher importers)
  • finished the first version of a multiple watcher. it is ugly and only for advanced users for now--please check release post for more information
  • added 'use the multiple watcher on DnD events' to options->downloading, which will send thread url drag and drops straight to a new/existing multiple watcher!
  • gui session load now 'shows' the first page but loads everything to the right in the background, which saves some CPU and memory for large sessions. this means it now starts focused on the leftmost page rather than the right--see if you like it or not
  • 'clear and load session' now fully deletes old pages and takes a little break between the clear and load step to make sure old large sessions are deleted tidily before loading the new stuff
  • cleaned up the new leaner popup message display code to react better to (and re-layout) sub-changes in its popups (so now when a subscription popup spawns a new progress gauge, it _should_ be positioned in the right place immediately)
  • if an import folder/subscription publishes files to a page while the main gui is minimised (and the action would result in a new page creation), the page add will be delayed until the gui is no longer minimised (new page layout in this case was broken due to minimised parent frame)
  • the edit subscription query panel will now spawn with the query text input focused
  • added a patch to ensure new booru import pages get query input focus after a second if the focus attempt on init failed (this will be fixed better in the new downloader system)
  • restored the 'clear deleted files record' button to the 'combined local files' service on _review services_, which was previously hidden during a significant service rewrite.
  • if critical master tables are missing on boot, the client will warn and provide info and then abandon the boot
  • if cache similar files tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway (this replaces some old repair code that wasn't always kicking in at a good time)
  • if cache autocomplete tables are missing on boot, the client will warn and provide info and offer to recreate and repopulate them
  • if mappings tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway
  • the emergency 'repair missing file locations' panel now runs in a more stable way
  • this emergency dialog will now also recognise 'I fixed all the file paths but cannot do the thumbnail paths' situations and will present the same 'ok, make sure you run regen thumbs after boot' dialog as if all the missing paths were thumbnail related
  • you can now turn off x/y page name import progress under options->gui
  • fixed an issue where tags that begin with ':' like ':p' were not getting through the new importing system
  • fixed an issue where tags that begin with ':' like ':p' could sometimes get prepended extra escape characters and end up looking like ':::p'
  • fixed sibling tag searching (e.g. if a->b exists and you search for b, results with a but not b should also appear), which the recent tag search optimisation accidentally disabled
  • whole bunch of refactoring and cleanup of ClientData and ClientGUICommon
  • cleaned up some spammy splash text status setting that was flooding some debug info with useless garbage
  • gave getting_started_tags.html's tag repo section and access_keys.html a pass, updating the ancient screenshot and linking advanced users to the latest QuickSync location
  • some misc help updates
  • misc autocomplete logic improvements
  • misc ui display fixes
  • misc wx destroy code stability improvement
  • updated ffmpeg for windows builds to the new 4.0 release
  • updated sqlite for windows builds

• version 306

  • the file import status list now has 'open selected import files in a new page', which should show up where it is possible. this is a bit prototype and ugly--it'll show _all_ files, including in-trash and permanently deleted (which will show up with the hydrus thumbnail)
  • the file import status list now prefixes the already in db/deleted notes with 'url' or the hash type that lead to the recognition
  • these redundant/deleted notes now also propagate up from 'during import' recognition phase as well
  • the 'delete seeds of type x' entries on the file import status button's right-click menu are now split into three smaller individual tyes and are more explicit about exactly which status types they will remove
  • like import folders, subscriptions can now optionally publish their files to pages as well as popup buttons. also, subscriptions can optionally publish their files separately for each query instead of all merged together
  • sped up multiple tag queries significantly
  • sped up simple (file size, mime, etc...) system predicate queries that also include a tag/namespace/wildcard predicate significantly
  • added a pixiv parser that pulls the japanese tags to the defaults--users can switch to this if they prefer under network->manage url class links
  • fixed the 4chan parser to get part of comment as backup subject/page title
  • removed the 'newgrounds' entry from the normal gallery page creation ui, as the basic gallery parser no longer works due to a dynamic loading change on their end. I hope to have it back with the new gallery parsing system I will soon be writing
  • the edit url classes panel now has a little text box to put in example urls and see which class, if any, that they match to
  • improved layout of edit url class links panel
  • all url types are now displayable in the media viewer--only post url classes are default on
  • the new (x/y) import page page_name progress count is now updated on all alterations to this value (previously, this was not updating when a user interacted with the import queue, only when the natural downloader loop cycled)
  • added 'can produce multiple files' option to post url url classes, which informs client url-checking logic whether the url can be relied upon for 'already in db/deleted' calculations
  • the pixiv file page url class now has 'can produce multiple files' checked, meaning some bad pixiv url association logic due to other sites referencing it as a source url is now fixed
  • added a 'twitter tweet' url class, which is also a 'can produce multiple files' post url
  • added a 'sync known urls?' action choice to the duplicate merge options panel, which governs whether urls should be copied from worse to better or in both directions
  • gave the edit duplicate merge options panel a layout pass
  • the edit duplicate merge options panel will now disable pointless/over-complicated choices on non-custom actions, let me know if this is a pain for your workflow
  • added a 'manual' web browser path override to the 'files and trash' options panel, which fixes the new share->open->in web browser option for Windows and also fixes some #anchor link propagation
  • consolidated all URL/Path web browser launching code to one location
  • 'open in web browser' is now available for non-advanced_mode users and the 'open' submenu of the share menu is available in the preview window and the media viewer
  • fixed a bug that was causing import folders to publish incorrect file identifiers, which was poisoning popup buttons and import page destinations
  • gui sessions that fail to load a page will recover and continue to attempt loading the rest of their pages. some popups detailing the page's serialised data and error will be presented
  • gui sessions that fail to save a page will recover and continue to attempt saving the rest of their pages. some popups detailsing the page's rough info and error will be presented
  • the core controller inside all media pages will now present itself in a more beautiful way when asked to dump itself to a log (which should beautify the above save error a bit)
  • wrote a subsidiary database->check->just repo update files that tests integrity of only repository update files
  • fixed an issue where default tag import options were sometimes not being saved from the new dialog in the networking menu
  • wrote a couple of layers of bad tag protection to help the new downloader deal with some occasional bad output from the old downloader
  • network jobs can now reattempt connection attempts up to three times on POST requests (if you ever got inexplicable immediate 'could not connect' errors on repository uploads, this should now be fixed)
  • replaced some archaic misc old import code with the new system, cleaning up a bunch of stuff and making space for further refactoring along the way
  • fixed tags blacklist not being inherited in the old (through options dialog) system
  • improved some invalid domain error handling
  • fixed an animation update issue that would pause naturally updating controls on non-main-gui frames when there were no regular media pages open on the main gui
  • added a BUGFIX option to 'files and trash' option page to override the default temp path for almost all client temp path requests
  • the minimum value for the 'vacuum period' in maintenance and processing options is now 28 days. the control also has a little explanatory tooltip
  • the 'try to auto-link url classes and parsers' function now always preferences parsers alphabetically
  • fixed a typo in the string transformations prettyfication code that incorrectly summarised 'take the last x characters' as the opposite
  • misc fix to file hash generation and status checking code
  • the 'export tags to .txt files' checkbox on the export files panel will no longer bother you with a dialog as you uncheck it
  • wrote some code to make it easier and more fool-proof to update the domain manager with new url classes and parsers on my end
  • improved some popup message manager ok-to-alter-ui logic when the main ui is minimised and so on
  • fixed some potential crash conditions (affecting linux mostly, seems like) in the service credential testing and access key fetching ui code
  • fixed a bug when 'stopping' a gallery parse during a long error pause (like when it holds on '404')
  • sped up some old set intersection code
  • some import file presentation refactoring
  • some url content application pipeline cleanup
  • misc cleanup

• version 305

  • fixed the pixiv url class, which was unintentionally removing a parameter
  • wrote a pixiv parser in the new system, fixing a whole bunch of tag parsing along the way, and also parses 'source time'! by default, pixiv now fetches the translated/romaji versions of tags
  • finished a safebooru parser that also handles source time and source urls
  • finished an e621 parser that also handles source time and source urls and hash!
  • wrote a danbooru parser that also handles source time and source urls and hash!
  • as a result, danbooru, safebooru, e621, and pixiv post urls are now drag-and-droppable onto the client!
  • finished up a full yiff.party watcher from another contribution by @cuddlebear on the discord, including url classes and a full parser, meaning yiff.party artist urls are now droppable onto the client and will spawn thread watchers (I expect to add some kind of subscription support for watchers in the future). inline links are supported, and there is source time and limited filename: and hash parsing
  • fixed some thread watcher tag association problems in the new system
  • when pages put an (x) number after their name for number of files, they will now also put an (x/y) import total (if appropriate and not complete) as well. this also sums up through page of pages!
  • if a call to close a page of pages or the application would present more than one page's 'I am still importing' complaint, all the complaints are now summarised in a single yes/no dialog
  • url downloader pages now run a 'are you sure you want to close this page' when their import queues are unfinished and unpaused
  • if the subscriptions for 'manage subscriptions' take more than a second to load, a popup will come up with load progress. the popup is cancellable
  • added a prototype 'open in web browser' to the thumbnail right-click share menu. it will only appear in windows if you are in advanced mode, as atm it mostly just launches the file in the default program, not browser. I will keep working on this
  • harmonised more old download code into a single location in the new system
  • created a neater network job factory system for generalised network requests at the import job level
  • created a neater presentation context factory system for generalised and reliable set/clear network job ui presentation at the import job level
  • moved the new downloader simple-file-download-and-import to the new file object and harmonised all downloader code to call this single location where possible
  • did the same thing with download-post-and-then-fetch-tags-and-file job and added hooks for in the subscription and gallery downloader loops (where a parser match for the url is found)
  • the simple downloader and urls downloader now use 'downloader instance' network jobs, so they obey a couple more bandwidth rules
  • harmonised how imported media is then presented to pages as thumbnails through the new main import object
  • the new post downloader sets up referral urls for the file download (which are needed for pixiv and anything else picky) automatically
  • improved file download/import error reporting a little
  • entering an invalid regex phrase in the stringmatch panel (as happens all the time as you type it) will now present the error in the status area rather than spamming popups
  • fixed a bug in the new parsing gui that was prohibiting editing a date decode string transformation
  • fixed enabling of additional date decode controls in the string transformations edit panel
  • added a hyperlink to date decoding controls that links to python date decoding explainer
  • if a source time in the new parsing system suggests a time in the future, it will now clip to 30s ago
  • misc downloader refactoring and cleanup
  • fixed an issue where new file lookup scripts were initialising with bad string transformation rows and breaking the whole dialog in subsequent calls, fugg
  • hid the 'find similar files' menu entry for images that have duration (gifs and apngs), which are not yet supported
  • added 'flip_debug_force_idle_mode_do_not_set_this' to main_gui shortcut set. only set it if you are an advanced user and prepared for the potential consequences
  • silenced a problem with newgrounds gallery parser--will fix it properly next week
  • fixed some old busted unit test code
  • rejiggered some thumb dupe menu entry layout

• version 304

  • renamed the new 'tagcensor' object to 'tagfilter' (since it will end up doing a bunch of non-censoring jobs) and refactored it into clienttags
  • attached a tag filter object to all tag import options to act as a tag blacklist. all tags that go through the import pipeline (except for a couple of old legacy instances) are now checked against the blacklist, and if a bad tag is found, the file vetoes! tag import options has some new ui to handle this and background code to deal with inheritance from defaults and so on
  • new file import urls that have url classes, no matter their source, are now normalised!
  • all new file import urls are now tested against both the original and normalised version of the url, so even though previously parsed urls remain un-normalised, new urls that are pre-normalised the same will not count as new! -fingers crossed-
  • on update, the db will get normalised copies of all existing urls. this means many files will now have two versions of its urls--some ui to collapse everything down to only the normalised version (after some human eyes have passed in front of this big change) will come in the coming weeks
  • some sites where normalisation is a consistent problem for later redownloads (like e621, which appends 'preview' tags to the post url) _should_ now be caught reliably!
  • the 'allow subdomains' on edit url class panel is now named 'match subdomains' and has a tooltip to better explain how it works
  • 'keep subdomains' is now 'keep matched subdomains' and has a tooltip as well
  • the 'keep matched subdomains' enabled behaviour (and some normalisation calculation) is now additionally governed by the 'associate url with files' value and api url conversion info rather than just 'match subdomains' and raw url type
  • fixed an issue that was stopping the 'associate url with files' option sticking in edit url class panel
  • edit url matches now resorts after an add or edit action
  • all listctrls with a wrapper panel now resort after an import from clipboard, png, or defaults call
  • url matches now match against www*. versions of their domain regardless of 'match subdomains' settings
  • updated xbooru url classes to prefer https
  • the manage url class links panel now has a 'clear' button to clear a url_class->parser link
  • introduced three new simple downloader parsers for yiff.party, thanks to @cuddlebear on discord for the submission
  • the old 'uninteresting mime' status has been expanded to a wider 'vetoed' status to represent all file imports that are abandoned without a particular error (e.g. tag blacklist, wrong filesize or resolution)
  • the import system now reports the total of 'num vetoed' as 'num ignored' in its summary statements
  • it now also reports 'num skipped'
  • the 'num successful' and 'num already in db' are now folded more neatly together in import cache summary statements
  • file downloads that are cancelled will now set a 'veto' state rather than a 'skip' state
  • improved file import exception handling across the board
  • improved how single-file-result parsing vetoes propagate up to the file import status cache
  • 404 network errors will now provide a 'veto' status rather than an 'error'
  • vetoes will not count as errors when deciding whether a subscription should be abandoned early (so a bunch of decomp bombs or 404s will no longer stutter a subscription!)
  • misc fixes and improvements to the new download stuff
  • wrote a new parsing cache that saves a lot of work in the new parsing system
  • improved the 'is this url known?' test to better deal with situations where all the given urls are galleries or unrecognised--a better aggregate of file status is formed, and 'already in db'/'deleted' statuses will apply if there is no evidence otherwise (the dev got the new logic for this from a legit nightmare about urls downloading over and over, so let's hope it works out)
  • the 'is this url known?' logic also recovers from 1->n url->hash relationships where it does not expect them, trying to find 'already in db' hashes over 'deleted' ones
  • to clear up some ambiguity, galleries or subscriptions now give a different 'checking in x seconds' status when waiting on the first page of a query
  • the 'noneablebytescontrol', as seen in edit file import options, will now correctly disable/enable its bytes sub-control when it is none'ed
  • a persistent issue with the new network engine sometimes failing to correctly error after certain broken connections (the computer going to sleep mid-download was a common cause here) should now be recovered from and the connection naturally reattempted
  • added three new shortcuts to the 'main_gui' shortcut set that allow for opening a new 'urls', 'simple', or 'thread watcher' downloader page
  • added two more shortcuts to 'main_gui' for new 'page of pages' and 'duplicate filter page'
  • moved some old 'new page' menu code to the new application command system
  • added numerous 'duplicates' shortcuts to the 'media' shortcut set that will work on selections of thumbnails
  • the thumbnail duplicates menu actions now go through the new application command system
  • fixed an issue where the current tag parents caches was not refreshing when notified
  • inputting a short invalid syntactic input on a 'read' tag autocomplete such as '-' will now clear the system predicates list--system preds should now only show on a completely empty input
  • fixed an issue where certain combinations of 'remove a tag, then re-add it' nullipotent actions in a single manage tags dialog transaction were not applying reliably (sometimes, the subsequent mirror action was not occuring due to a processing re-order optimisation at the db level)
  • made some animation code a little safer and quieter as a test for some users who were getting blitzed with some deadwindow error spam in certain situations--let's see if this changes anything
  • replaced all the em dashes in the help with double hyphens as github pages was rendering them wrong
  • added CrystalDiskInfo recommendation to 'help my db is broke.txt'
  • misc cleanup

• version 303

  • file post url classes can now be linked to parsers!
  • the 'raw url' downloader is now just the 'url' downloader. if a dropped url is matched as a 'file post' url and links to a parser, it will download it with that parser and default tag import options
  • url drag and drop now recognises file post urls that have linked parsers and will send them on to a url import page, just like 'raw' urls! they will use the new parsing system to parse tags and known urls and all that
  • wrote a way for the new download system to store and edit default tag import options--this can now be found under network->manage default tag import options, although it does not do much yet. eventually the old options will be transferred here, and the different downloaders will consult it more
  • refactored and cleaned up how some default tag import option calculation is done
  • the fixed 5 second wait time between gallery page fetches is now editable for downloader pages and subscriptions under options->downloading. subscriptions default remains 5s, gallery default is now 15s
  • the gallery downloader will present its 'x urls found so far' string in a more uniform and reactive way. it will also count down until the next page fetch
  • subscriptions will now similarly react quicker while the gallery parse step is ongoing, and it will count down to the next page fetch
  • the manage subscriptions dialog will now note in BIG RED TEXT if subscriptions are currently globally paused
  • subscriptions now consume less parallel-timeslot overhead, meaning they sync much faster when they have no work to do (which is most of the time)
  • separating subscriptions is now more sophisticated--if you wish, you can now choose just to separate a subset of the large subs's queries, and if you do so, you can decide whether to merge the extracted subs into a new larger sub or just have them as individuals. also, you will be asked for name/base_name for what to name the new subs
  • since separating subs is more complicated, the button is only enabled when a single large sub is selected. please do your separating one sub at a time!
  • merging subs will now ask if you want to rename the merged subs as it goes
  • refactored a giant heap of duplicate import code into single locations in the new file import object--things like applying tags and generating pre-import status are all now done in one place. a number of weird behaviours (like not applying known url associations in certain circumstances) should now be a lot better and unified
  • the way the db and other import testers handle whether a pending import is new or unknown is simplified
  • optimised the way the importers figure out whether to display a new import
  • the 'has the client seen this url before?' test used to figure out whether to download a file now explicitly ignores anything it isn't certain is a single-file File URL or Post URL (according to current url classes). associating 'gallery' known urls is no longer catastrophic to this system
  • the simple downloader now associates the entered url with the files subsequently imported
  • file import objects working in the new parsing system are aware of the new 'should this url be associated' option and will be given gallery urls to hold on to as appropriate
  • url classes now have an explicit option as to whether they should be associated with files imported from them, which is not appropriate for dynamic CDN File URLs but is appropriate for multi-image tweet links (which will likely be Gallery URLs in the new system), for instance
  • general importer code cleanup across the board
  • threw together some early ui to show current jobs in the network engine under network->review network jobs
  • popup messages will now try to set a max width of roughly 56 characters wide rather than 400 pixels. this value is editable under options->gui
  • updated the listctrl in the export files frame and the way this frame generates and stores export filenames. let's see if linux users who had crashes with this have better luck
  • refactored stuff out of the bloated ClientImporting file into seperate new files
  • refactored client network bandwidth and session code out to their own files as well
  • refactored client network objects to its own file
  • cleaned up some open externally debug code linked to callto report mode
  • a common grid layout used across the program will copy control tooltips to the string on the left
  • removed the 'downloader' network context type, which was never fully introduced and wasn't turning out to offer much in the new bandwidth tracking system

• version 302

  • improved how simple downloader parsing formulae are stored and passed around
  • the edit simple downloader parsing formulae panel now has an 'add defaults' menu button
  • the simple downloader formula edit panel handles its buttons better and can now do import from/export to clipboard/png and reimport from the defaults, and fixes duplicate names automatically
  • simple downloaders now remember their current parser through a session save/load
  • simple downloaders will set a new default parser selection for new simple downloader pages on any choice event
  • renamed all the default simple downloader parsers to more clearly explain what they do
  • added 'all files linked by images in page' to the simple downloader, which does the old page of images' behaviour, and is hence pretty decent for most imageboards
  • added a simple mewch thread html parse formula to the simple downloader
  • added a very simple webmshare parse formula to the simple downloader
  • added 'imgur image album' to the simple downloader, which will fetch the first 'page' of results from an image album. full parsing will have to wait for the gallery downloader update
  • subscriptions can now run simultaneously! you can set the max number at options->downloading page--the default remains 1, so default behaviour will not change
  • if subscriptions guess they can eat some bandwidth in the next 30s but it turns out they can't, they will bandwidth-override that last job after the 30s expires. this will stop some edge cases from causing subs to potentially hang for hours just on one last request
  • 'explicit tags' are now renamed to 'additional tags'
  • you can now set media to initially scale to 100% even when it is larger than the media or preview canvas
  • the html parsing formula can now 'ascend' as well as search down the DOM tree--so, for instance, you can find all 'img' below an 'a' and then jump back up to the 'a' to fetch the 'href' of that image link!
  • html parsing formulae can now search without a tag name--so, for instance, 'find all tags with class = "whatever"' is doable, without having to specify div or span etc..
  • the html parsing formula rule panel is hence a bit richer. it also presents a preview of the rule's string as you edit
  • the client no longer needs lxml to boot, and if html5lib is present, it will prefer that for html parsing. if both libraries are missing and html parsing is requested, it will raise an exception explaining the error
  • the builds now include html5lib, which is better at recovering from some bad markup than lxml, and as a result some parsing jobs that formerly did not work (such as 'akaiha_(akaihasugk)' on danbooru) now do
  • rewrote how services test their functional status to better account for different states. the logic here should be a bit better now--repositories will still be able to process if their network side is down, but everything will pause if any pause is set, and it should all propagate up to higher levels of error catching and status reporting better
  • hydrus can now deal with really big decompression bombs (>~180 megapixel)
  • filtered out en-dashes (as an alternative to standard hyphens in a "? + -" link-prefix) from booru tag parsing
  • fixed a string generation issue that was stopping date decode string transformations from summarising themselves
  • fixed some catastrophic boot failure gui reporting
  • cleaned up a double-exit error on dialogs that could occur when spamming exit events with a script like AutoHotkey
  • improved some dead cpu thread clearout code
  • updated misc old code
  • misc cleanup

• version 301

  • after discussions with Sankaku Complex about their recent bandwidth problems, added a new 64MB/day default bandwidth rule for sankakucomplex.com--please check the release post for more information
  • the 'page of images downloader' is now called the 'simple downloader' that uses the new parsing system (particularly, a single formula to parse urls)
  • the simple downloader supports multiple named parsers--currently defaulting to: html 4chan and 8chan threads, all images, gfycat mp4, gfycat webm, imgur image, imgur video, and twitter images (which fetches the :orig and also works on galleries!)
  • there is some basic editing of these parsing formulae, but it isn't pretty or easy to import/export yet
  • the new parsing test panel now has a 'link' button that lets you fetch test data straight from a URL
  • added a 'gather to this page of pages->dead thread watchers' menu to the page of pages right-click menu--it searches for all 404/DEAD thread watchers in the current page structure and puts them in the clicked page of pages!
  • cleaned up some page tab right-click menu layout and order
  • fixed tag parents, which I previously broke while optimising their load time fugg
  • the new favourites list now presents parents in 'write' tag contexts, like manage tags--see if you like it (maybe this is better if hidden?)
  • sped up known_url searches for most situations
  • fixed an unusual error when drag-and-dropping a focused collection thumbnail to a new page
  • fixed a problem that was marking collected thumbnails' media as not eligible for the archive/delete filter
  • wrote a 'subscription report mode' that will say some things about subscriptions and their internal test states as they try (and potentially fail) to run
  • if a subscription query fails to find any files on its first sync, it will give a better text popup notification
  • if a subscription query finds files in its initial sync but does not have bandwidth to download them, a FYI text popup notification will explain what happened and how to review estimated wait time
  • delete key now deletes from file import status lists
  • default downloader tag import options will now inherit the fetch_tags_even_if_url_known_and_file_already_in_db value more reliably from 'parent' default options objects (like 'general boorus'->'specific booru')
  • the db maintenance routine 'clear file orphans' will now move files to a chosen location as it finds them (previously, it waited until the end of the search to do the move). if the user chooses to delete, this will still be put off until the end of the search (so a mid-search cancel event in this case remains harmless)
  • the migrate database panel should now launch ok even if a location does not exist (it will also notify you about this)
  • brushed up some help (and updated a screenshot) about tag import options
  • fixed a problem that stopped some old manage parsing scripts ui (to content links) from opening correctly
  • improved some parsing test code so it can't hang the client on certain network problems
  • misc ui code updates
  • misc refactoring

• version 300

  • wrote system:known url to find files that have--or do not have--certain types of urls. it works but is still a little slow--I can optimise it later!
  • added exact match, domain, regex, and url class search types for system:known url
  • added a button to the top media viewer hover window that will start a file export drag and drop event if dragged from
  • moved the autocomplete dropdown results list down into a paged notebook
  • wrote a new 'favourites' page tab for the autocomplete dropdown results
  • hitting left or right arrow keys on an empty text input will move between the results tabs
  • hitting arrow up/down/page up/down/home/page or passing mouse scroll events will now go to the current selected page
  • typing regular search text into the input will automatically return the current page to the search results list
  • moved the 'tag suggestions' part of the 'tags' options page to a new page
  • added 'tag favourites' to the 'tags' options page to edit which tags show in this new tab
  • added import/export buttons to the tag siblings and parents dialogs. they'll export to clipboard or .txt file, and import from the same with an additional option to add_only (i.e. to not delete/petition conflicts with the existing list)
  • added some quick-and-dirty 'set as alternates/same/notdupes' buttons to the duplicate filter, which will quickly apply that status to the dupes and show some more dupes
  • sped up db loading time of tag siblings and parents significantly
  • added a short delay check to tag siblings/parents regeneration so rapid regenerations (such as when processing certain admin-side petitions) can be merged
  • fixed an issue where similar_to searches could return results not in the current file domain
  • fixed some spinctrls that were sizing to thin
  • fixed a bug in the manage server services dialog that was incorrectly dealing with port conflicts on edit service dialog ok
  • added a clientside and serverside assertion to test that all the services on a serverside modify services call have unique ports
  • fixed an issue where hydrus network services without access keys would sometimes try to sync their accounts (this was messing up some admin server setup)
  • fixed some misc dialog window structure
  • messed around a little with how the autocomplete dropdown hides and shows when in float mode--I _think_ it will now be less flickery and will otherwise position itself and receieve focus better
  • converted the 'export files' dialog to the new sizing system and also made it non-modal (i.e. you can now interact with the rest of the program while it is open)
  • wrote a more rigorous force-fit-all-tlws command to the debug menu
  • misc fixes
  • misc refactoring

• version 299

  • wrote ui to review and even edit session cookies by network context. it is still a bit rough but will help with future development.
  • added a 'open_selection_in_new_page' shortcut to the 'media' shortcut set that will work on the thumbnail view
  • added a 'export_files' shortcut to the 'media' shortcut set that will work on the thumbnail view
  • fudged manage siblings logic to not do the borked 'hey, that sibling already exists' as soon as you type the old sibling--it will now auto-petition any existing siblings when you click 'add' with a good automated petition reason that makes sense to the janitor
  • manage siblings now also only shows rows appropriate to the current selection like parents does. it gets a new 'notes' column to specify conflicts that will be auto-petitioned as above
  • manage siblings and parents now have a laggy 'show all pairs' checkbox to let you quickly review everything like you used to
  • fixed a database-level bug that meant petitioned and pending the same left-hand sibling tag (like petition a->b, pend a->c) would sometimes not both save together
  • when you middle-click or right-click->open a new page on a selection of tags/search predicates, the new page will now be named after the tags
  • if subscriptions hit their periodic file limit, they will now give a little popup message describing what happened and possible causes and actions the user can take
  • added a 'make a modal popup in five seconds' action to help->debug
  • modal popups will now hide/show other child frames (like review services) rather than minimise/restore, as this latter action can raise the entire progam to the front
  • added a 'make a parentless text control dialog' debug entry to test some key event catching
  • added a 'layout all tlws' debug entry to help->debug that'll hopefully help figure out some child window sizing/position issues
  • added a 'shortcut report mode', which will report caught shortcut keys and their matched commands, if any
  • fixed a little shortcut catching bug in the main gui
  • finished adding the new bytes control
  • updated ffmpeg for windows
  • improved sankaku default bandwidth rules to stop a subscription bandwidth rules mismatch that could sometimes make for subscription delays--existing users may like to add a 2GB/day rule for sankakucomplex.com
  • improved some service and account error handling to better propagate the exact problem up the exception handling chain
  • the clientside 'update A actually had hash B' repository sync error is now dealt with in a less severe way, and the bad update file is saved to disk with a request for it to be forwarded to hydrus dev for further investigation
  • file parsing is now more resistant to invalid negative values for properties like width, height, and duration
  • improved some focus code that may have been affecting linux stability
  • improved a deviant art url-not-found error
  • improved a wx version boot test
  • misc fixes

• version 298

  • wrote a new 'bytescontrol' and related noneablebytescontrol that allows for a uniform way to choose a bytes size value with a wider range--and deployed it all over the place
  • hitting a mapped shortcut for 'manage file notes' should now ok the 'manage file notes' dialog as well as open it
  • the manage file notes text control will now start with its caret at the end of the document
  • manage siblings/parents now state how many pairs they have
  • as an experiment, manage parents now only lists pertinent information in its listctrl. this is hacky but potentially a big improvement to the workflow here, feedback would be appreciated
  • added a new menu entry, 'network->pause->all new network traffic', which will indefinitely pause any new network jobs. this value will persist through a restart
  • 'pause subscriptions' is moved from 'services' to 'network'
  • 'pause import/export folders' is moved to 'file' and all the import/export folder stuff is sent down to its own submenu
  • in lieu of proper session inspection gui, added some debug 'reset login' entries to the network menu for pixiv and hf
  • if a page has files but none are selected, it will now say the total size in the status bar
  • the edit bandwidth rules control and its subsidiary dialog use some saner and more user-friendly layout and presentation
  • the previous search distance on the 'review bandwidth usage' frame is now remembered
  • fixed some bad logic where a 'copy_bmp' event could trigger despite the current media being None or a non-static image
  • you can no longer open multiple copies of the subscriptions or import/export folder manage dialogs if you hit the menu entry multiple times while the first is waiting for the jobs to finish
  • like import folders and subscriptions, manage export folders now waits for currently running export folders to quit before opening
  • export folders run lighter and quit faster on client shutdown
  • the domain manager will give a better error if a URL submitted to it lacks a schema (the http or https part)
  • fixed a bunch of unicode error handling
  • fixed an issue with similar files metadata orphans prohibiting new file imports
  • finished up a simple shared shortcut processing object and replaced most old temp duplicate shortcut code with it
  • misc ui cleanup
  • misc text rendering fixes
  • misc shortcut code refactoring

• version 297

  • finished a prototype 'file notes' system. thumbnails and media viewer canvas now support 'manage->file notes' in their right-click menus. this launches a simple text box which will save its contents to db
  • added 'manage_file_notes' shortcut to the 'media' shortcut set
  • tag summary generators now have a simple show/hide checkbox and (for thumbnails) custom colours for background and text including alpha channel!
  • fixed a variety of timing and display logic related to subscription query DEAD vs next check time calculation
  • all currently dead subscription queries will be revived on update, just in case they were formerly set dead by accident
  • the 'fetch tags even if url known and file already in db' option is moved from the download/subscription panel's cog icon to tag import options
  • cleaned up tag import options layout, controls, internal workflow, and help button
  • added 'select all/none' buttons to tag import options panels with multiple namespaces
  • if a subscription is blocked by bandwidth, the manage subscriptions dialog will display that in its 'recent error/delay' column
  • the edit subscription dialog will show similar bandwidth blocking info, on a per-query basis, under a new 'recent delays' column
  • the review bandwidth usage panel will no longer show some unusual results by default that you can see with 'show all' hit anyway
  • the review bandwidth usage panel will show the usage at the current search distance in a new column
  • the review bandiwdth usage panel will show number of requests after data usage. this might be info-overload, so I might alter the syntax or roll it back entirely
  • fixed an issue with hentai foundry parser pulling images placed in the image description area instead of main image. this particularly affected the artist 'teku'
  • tags for deviant art and tumblr and thread watchers, which were formerly stored in volatile session memory--meaning half-completed import queues were losing their tags through a program restart--are now saved to the new import object directly
  • removed all the old volatile session memory patch code
  • added the new import object through a larger part of the parsing pipeline
  • deleted the old remains of the giphy parser--if it comes back, it'll all be rewritten in the new system
  • harmonised some other import pipeline code to the new system
  • added a new 'management and preview panels' submenu to the 'pages' menu
  • added an option to control 'save sash positions on close' to this menu
  • added an entry to force-save the current sash positions to this menu
  • added an entry to 'restore' the currently saved sash positions to all pages to this menu (this is useful if your window resizes real small and all your pages get crushed up)
  • rejiggered how URL Classes are matched with URLs to make sure some Post URLs are not lost (this was affecting Hentai Foundry Post URLs, which were sometimes not displaying in the media viewer despite matching)
  • fixed an issue where the duplicate filter page's jobs would not trigger an update after a job finished
  • fixed an outside chance of a crash after running a duplicate filter page job
  • improved how strings are coerced to unicode--now the preferred system encoding will be tried before utf-16, which should improve support for characters in various non-unicode sources (like neighbouring .txt files)
  • fixed an issue with the client's local booru and flash files (and some other file fetching and mime reporting is a bit faster and neater overall)
  • the options should be more reliable about redrawing all thumbnail banner summaries on an option ok now
  • the options->media->media zooms option will now remove any <=0.0 values when it saves
  • fixed up some old test code
  • improved how some thread-to-gui update reporting code works
  • deleted some old network object code
  • converted manage subscriptions panel to an edit panel--a decoupling refactor I will likely ultimately make across the program
  • wrote a help page for content parsers
  • did the first half of a help page for page parsers
  • misc refactoring
  • misc cleanup

• version 296

  • the 'allow decompression bombs' option is now moved to 'file import options'. it defaults to False
  • file import options now allow max size and max resolution rules. they default to None
  • file import options now allows a max gif size rule to deal with THE SFM COMMUNITY. it defaults to 32MB
  • file imports will give better quality errors if they fail due to file import option exclusion rules
  • file import fail errors will now also specify the incorrect amount--e.g. if a file is 64KB but the minimum limit is 256KB, it will state this in the error
  • wrote a bunch of unit tests to make sure the new complicated file import options object works good
  • finished the tag summary edit ui and added a new 'tag summaries' page to the options
  • you can set the thumbnail top, thumbnail bottom right, and media viewer top summaries here, and everything should link up right!
  • the old options to hide thumbnail top and thumbnail bottom right text are removed from the options->gui page
  • tag summaries now include pending tags again
  • tag summaries will order tags according to hydrus numeric tag sorting rules (0 < 0a < 0b < 1 < 9 < 10)
  • tag summaries will collapse namespace groups that all start with decimals into a "min-max" phrase instead of the whole list (so 'collecting' a chapter will list p1-23 for the page phrase instead of each number in turn)
  • fixed some potential error popups from the autocomplete dropdown when not in float mode (this hit non-windows users more often)
  • split the parser help pages up a bit more and updated the client ui html links
  • finished off the formulae help pages
  • updated to wx 4.0.1 on all platforms. it promises increased stability!
  • default tag import options are now moved to the 'importing' options panel, alongside the default file import options
  • the database will deal with and recover from rollback exceptions more reliably. rollback-loops should be fixed, but the original event will now also produce a request to restart the client (which will safely reset everything) anyway
  • improved how I create and layout the 'help for this panel -->' help buttons
  • the job scheduler now regularly checks for and clears out dead wx-related jobs (this saves a bit of memory)
  • cleaned up an old tag retrieval method that was wasting CPU. some larger pages with many thousands of tags should do tag calculation after selection and so on a bit faster
  • fixed a thread shutdown test that wasn't working until app shutdown time
  • improved some more thread shutdown code
  • deleted some old thread-inefficient calllater and wx-safe timer code
  • emergency server errors that occur during the response writing stage should be handled better by the client
  • emergency server errors are more robust in general serverside
  • improved the quality of the errors thrown by the new network engine's login system
  • improved some test timing and key press simulation code
  • misc refactoring
  • misc improvements

• version 295

  • fixed the runtimeerror popups that would come up on restore from minimise or main gui move after the complete destruction of a general search page
  • cleaned up some main gui move code generally, and removed a memory leak on the way
  • file queries can now cancel at multiple checkpoints during the first phase, saving a bunch of CPU time on certain large queries that are replaced mid-search
  • after a file query has been going three seconds, a little 'stop' button will appear beside the regular autocomplete input. clicking this will cancel the current query! it will stop when it next hits one of the checkpoints above
  • the floating autocomplete dropdown should be less flickery in some circumstances
  • dejanked some more file query code
  • added a 'clear orphan file records' entry to the database->maintain menu. this looks for and purges orphan file rows as you may have seen a notification about recently. this mostly affects the duplicate filter system
  • fixed up the delete file code to be a bit more robust--it should lead to fewer orphans in future
  • all the parsing edit panels have new layout: they no longer have info panels but instead a help button that points to the html help, and the edit and test panels are now beside each other rather than in notebook pages
  • harmonised a bunch of the parser ui test panel code, refactored how the results are stored
  • the test panel now presents a better 'preview' of what it contains (the actual text control has like 64KB text limit on some OSes and has unreliable text encoding rules, so using it as the raw container for the example data has lead to problems), and we now read and write the example data with a couple of new copy to/paste from clipboard buttons
  • wrote another new test panel for subsidiary page parsers that does the separation formula stuff a bit better. the test results now come back for all posts as well, rather than just the first
  • added a new 'deeply_nested_dialog' frame key to options->gui for the parsing ui to better lay out five or six nested dialogs in a nice 'topleft' way
  • the 'topleft' frame padding is reduced from 50 to 24 pixels to better fit in deeply nested dialogs
  • misc parsing ui improvements and little fixes
  • the manage url classes and manager parsers dialogs now have a better 'add defaults' button that allows you to just select the defaults you want (by name) from a checklistbox
  • wrote a parser for 420chan and added it to the defaults. it should automatically add and link up when you update
  • if the install_dir/db directory is not writable-to (e.g. you have installed the program to a protected location like "C:\Program Files"), the client and server will default to ~/Hydrus as the db directory
  • wrote a new 'TagSummaryGenerator' class that will do 'bunch of tags'->'nice summary string' conversions for the thumbnail banners and export filenames
  • substituted some static tagsummarygenerators to do thumbnail banners
  • did the same for the media viewer top-center namespace summary
  • started some edit ui for tagsummarygenerators--I'll have some proper customisable stuff in the near future
  • moved the background memory maintenance and misc daemons to the new job scheduler, reducing thread count and idle CPU some more
  • added a debug 'show scheduled jobs' entry for the new job scheduler system
  • decompression bomb failures will no longer count towards a subscription's fail count, so having a bunch of them won't abandon a sync
  • fixed and otherwise improved a potential crash condition when a thumbnail panel closes while a menu is popup'd on it
  • to forestall this program instability, the thumbnail window will no longer replace while its menu is open. the behaviour after this delayed window delivery is slightly borked, but it isn't a crash so I'm ok with it for now
  • removed some other jank from the thumbnail media panel swap code
  • non-cancellable modal popups will no longer have the 'close' button. trying to close them with the dialog's X button will still give the 'sorry lad, can't cancel' error
  • rating and file service system predicates for services that no longer exist will now render a neat 'unknown x system predicate' presentation string rather than throwing an error
  • searches in 'all known tags'/'specific tag domain' no longer provide system:untagged, wew
  • some delayed events are now posted in a more thread-safe way
  • misc refactoring

• version 294

  • fixed video scan
  • fixed up some import folder logic--they now run 'look for new files' checks separate from 'import anything still in the queue', so they can now catch up on outstanding files more easily
  • the ten minute file-processing break is reverted--import folders now just 'save' every ten minutes, to forestall lost work on a crash
  • import folders now have an explicit 'check regularly' checkbox to control whether you want it to check regularly or only when you tell it to. the paused status now means 'do not ever work'
  • import folders now make a 'working' popup, like subscriptions do, that shows new file discovery progress and found file import progress. it has a cancel button that will stop the current job and can be hidden in the import folder's options. existing import folders will default to ON for this, so you'll likely see one of these right after you update
  • import folders can now publish their files to a new page! this effectively cuts out the button step. furthermore, if a page already exists with the import folder's name (e.g. a previous page the import folder created is still around), the import folder will publish the new files to _that_ page, updating it
  • file popup buttons with the same name will now 'merge'! multiple work cycles of a subscription or import folder will now just update the first button rather than spamming several
  • running import folders will respond faster to a client shutdown event
  • wrote some better controller-level thread management, including surplus thread deletion--idle CPU should be reduced on import-busy clients
  • wrote some code to deal with sub-second times a bit better in certain places
  • finished off a new scheduled job queue that collapses an old multiple-thread-when-idle system down into just one
  • many things are moved to the new job scheduling system--all the old calllater calls, and the popup message timers as well
  • completely removed the old wx timers the autocomplete input was using, as they were just too much of a hassle. any crashing these were causing is now gone--it all works on the simpler scheduled job queue now
  • the autocomplete dropdowns are better at judging when to engage their timers and so now use far less idle CPU
  • fixed a crash that could occur sometime after starting a duplicate filter maintenance task from the dupe filter page
  • fixed several unlikely but plausible crashes on the admin-side of petition processing
  • wrote a bunch of help for the url classes and new parsing system--it isn't finished yet, but then neither is the new system!
  • if the client fails to initialise the db, it will now try to present the error in a bit of screenshottable-gui before the program quits
  • improved thread watcher error handling when the given url is unwatchable
  • lots of timer related cleanup and tiny fixes
  • misc fixes

• version 293

  • fixed the issue with options dialog not opening again after a save--I apologise for the inconvenience
  • the default system:time imported predicate (as set in options) is reset to the default '< 7 days'
  • fixed another potential crash with the manage tags dialog (in fact with any dialog that has an embedded autocomplete or any other autocomplete with a non-floating dropdown list, such as the import file path tagging dialog)
  • the sankaku user-agent has been updated to be a generic Firefox string "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0", which seems to work with their new rules. if your sank has been broken and you haven't touched the settings that do this stuff, your sank should be magically fixed on update
  • the default global hydrus user-agent has been updated to the more polite and compatible "Mozilla/5.0 (compatible; Hydrus Client)". if your previous global user-agent is currently 'hydrus client' (the old default), you'll be updated
  • import folders now deal with interruptions more skillfully and take a break and save their progress every ten minutes
  • the manage import folders dialog now waits properly for any currently running import folders to quit and save themselves before opening itself (like how the manage subs dialog does it)
  • import folders deal with some error states in a better way and are able to cancel out of their loops more reliably when an error occurs
  • added a new checkbox option to options->tags that lets your 'write' autocomplete inputs (ones where you can submit new tags, like in manage tags) select the first result with non-zero count by default. this means it will skip over the first result if it is just nonsense like 'boyp' that you typed to get the results

  • linked all the final behind the scenes stuff in the thread watcher together with the new parsing system. this stuff remains complicated and non-user-friendly, so please feel free to completely ignore it and the following batch of points

  • wrote new url classes and parsers for 4chan and 8chan--they should all be automatically linked up on update
  • the new thread parsers will use the first part of the first post's comment if no subject is set
  • the domain manager can now cope with URL->API URL links
  • manage url class links now displays the current expected api pairs of URL class links and filters out what cannot be currently parsed as a result
  • manage url class links now only displays watchable urls in the bottom listctrl and permits parsers to be linked!
  • the domain manager can now try to link url classes with parsers based on example urls!
  • a button to fill in missing links is now available on the manage url class links panel
  • content parsers that produce urls now define different url types--file, post, and 'next gallery page'
  • content parsers can now parse a 'source timestamp' type
  • veto content parsers now use a StringMatch object to do their matching. existing veto content parsers will be updated and _should_ work about the same as before (unless the previous string was crazy in regex rules) using a regex match
  • html formulas now pick up the 'string' value more reliably in complicated or mangled html
  • html and json parsing formulas can now work without any rules, in which case they action the top level node (this is useful to replicate the json/html for subsidiary page parsing)
  • activated some of the new file import objects' features for the new parsing system--now, they store tags and known hashes as informed by what is parsed. this information is tested and used in import
  • the db can now test all the hash types--md5, sha1, sha256, sha512--during pre-import checking
  • the new parsing system now takes a temporary 'parsing context' that will eventually be able to receive and serve temporary variables but at the moment just holds a 'url' entry if the parser would like to use the url used to fetch the data anywhere
  • all the new parsing ui now has a button to edit the parsing context, and the example parsing context is saved to the topmost page parser object
  • wrote some url classes for 420chan.org--they will be added on update
  • fixed an issue with the 4chan thread url class
  • the standard new listctrl panel wrapper can now now provide a special 'export to pngs' for certain named objects that will quickly export one object per png
  • the 'import from png' dialog as launched from the standard new listctrl panel wrapper now allows multiple png selection
  • the default parsers (and now the url classes as well) are stored in install_dir/static/defaults. they are read from here when you create a db or ask to load from defaults, so feel free to play around with this
  • the manage services dialog now has an additional Yes/No warning/confirmation dialog if it expects to delete any services on dialog ok
  • page tabs escape mnemonic characters (typically '&', which in ui labelling rules makes the following character an underlined shortcut) more reliably
  • in an attempt to stop subs making so many separate small file popups, the subscription daemon will now only naturally check every four hours (previously one hour) and will wait up to ten minutes for bandwidth to free up before dumping out (previously 30 seconds). these values will be user-configurable in a future update
  • fixed import for some unusual apngs
  • corrected some more GMT/local time stuff with the new system:time imported 'date' mode
  • misc GMT/local time fixes (profile logs names themselves correctly, etc..)
  • improved how the thread watcher reports thread-check errors in unusual situations
  • fixed an issue generating network contexts when the initial domain is 'localhost'
  • improved some misc program stability
  • started a new job scheduler object to reduce idle thread count and CPU usage--will play around with it a bit more and see about integrating it in the coming weeks

• version 292

  • extended system:age to support searching by fixed calendar date, e.g. "system:age > 2018/01/24"
  • 'system:age' is also renamed to 'system:time imported' and presents nicer strings
  • I believe I have fixed the manage tags dialog crash
  • the manage tags dialog now loads related tags, file lookup script info, and recent tags off the main ui thread, meaning the dialog itself should launch much faster in almost all situations
  • cleaned up how a bunch of autocomplete<->manage tags events occur
  • 'set_search_focus' works in the manage tags dialog again, drawing from the new shortcut system
  • fixed ctrl+scrollwheel on autocomplete entry, which now correctly scrolls the selection over the results again
  • cleaned up a bunch of the taglist scroll and selection code, particularly for tag children/parent selection events
  • fixed a potential crash bug initiated by en early exit from the export files dialog
  • fixed a potential crash bug initiated by an early petition processing page delete
  • fixed a potential crash bug initiated by an edit repository close during access key fetching
  • the thumbnail view should be a bit less jittery--the scrollbar should fade neater on mouseover, and maybe the thumbs are smoother as well (I turned off a now-superfluous double-buffering mode, please report any visual bugs)
  • added 'check_all_import_folders' to main_gui shortcut actions
  • menus off the main menubar that are out of date (this often happens to 'pages' after a session addition/delete) will now be disabled (greyed out) until they are back in sync
  • the duplicate filter now detects media it cannot display more carefully. if it runs across undisplayable media due to file orphan status, it will skip over it. if there is nothing else to display, it will ask the user to inform hydrus dev
  • added json support to the parsing engine!
  • wrote a new json parsing formula object
  • wrote a new 'compound' parsing formula object that combines multiple formulae together
  • wrote new panel for editing formulae with some cleaner button workflow
  • added multiple 'separated' content parsing solutions to page parsers and wrote all new gui to reflect that
  • expanded the new separated content parsing solutions to be full-blown nested sub-page-parsers, wew
  • put some more general work into the new parsing engine--the new page parser object now saves and serialises, so it can be imported and exported. I don't expect to change it much more, but don't rely on that! this is all still under construction
  • added 'file hash' content type to content parsers and improved ui support for when string conversion rules generate bytes data etc...
  • new parsing objects now render their parsable content correctly in the 'produces' column
  • misc cleanup of new and existing parsing dialogs
  • fixed the handling of escape/cancel events on the 'select from a list of strings' dialog
  • on update, users with a certain kind of orphaned file will be informed of the numbers and a request to inform hydrus dev
  • fixed exporting serialisable object to png, which was failing due to a bmp data change in the new wx
  • fixed a couple of places where I was switching GMT and local time around
  • deleted some old useless tag editing and commandevent code

• version 291

  • expanded the permissible shortcut keys to all reasonable ASCII characters, such as comma and ampersand and the types of brackets
  • fixed a delayed crash sometimes caused by the file import frame (the one launched by dropping some files on the client or going file->import files)
  • the file import frame does its file parsing job in a generally more sensible way, particularly when new drops occur while parsing is ongoing
  • fixed a crash that could be initiated by the manage tags dialog setting the favourite tags late
  • fixed a crash that could be initiated by manage parents closing before results came in
  • fixed a rare possible crash that could be initiated by a review service panel delivering results late
  • when media pages load, they will be more thread-crash-polite to wx
  • wrote a new 'PageParser' object that will be the primary parser in the new downloader engine. this can deal with a variety of complicated parsing situations. it is still a work in progress, not yet ready for proper use
  • started new 'manage parsers' dialog for the new parsing system and arranged a bunch of things around that to make the eventual transition easier
  • edited some more old 'parsing scripts' ui to again make the transition easier. file lookup scripts will be converted to something new as we move to the new parsing system
  • fixed some bugs in contentparser instantiation
  • fixed a bug in the html tag rule edit dialog that wouldn't allow attribute pair editing
  • the names in the manage tags file lookup script dropdown are now sorted
  • removed some old redundant error-hiding code from the pubsub system
  • a common 'is my page closed' check is now a lot simpler and faster
  • fixed an edge shutdown bug where log messages were not able to be written to the log file
  • some media viewer objects delete themselves more safely
  • maybe fixed a rare menu refresh bug in linux
  • improved how hydrus deletes temp paths--if they are 'in use by another process', it will now put them aside and try again later. this error typically occurs when file mimetype parsing fails, and due to the error context hanging around, an open file handle is still in scope.
  • the client is better about recovering from missing tags in the db--in certain circumstances, it will now generate a 'unknown tag:random hex' as a simple placeholder that can be better used to investigate the error, rather than dumping out
  • downgraded os x opencv to 3.1.0 to hoyefully increase compatibility
  • the server closes a little more neatly
  • upnp errors are more verbose
  • misc code cleanup

• version 290

  • content (tag and rating) shortcuts in the 'media' shortcut set now work on the thumbnail view!
  • tags in lexicophically sorted taglists will now sort numerically if their subtags start with decimals (e.g. page:3 will come before page:20 now)
  • trying to add/edit a subscription query that is a dupe will now give you a messagebox saying no, and pasting multiple queries will filter out the dupes and inform you (so, if you aren't sure if some of your list of ten artists to add is already in your big 'artist sub', just paste them anyway--it'll take out the dupes for you)
  • string converters now support regex substitution! (I expect to eventually integrate this into the filename tag parsing panel!)
  • 'modal' popup messages (that create a new dialog and stop interaction with the program, like the one launched for repository processing) will now minimise non-main-gui frames while they work. these frames will be restored when the job is done (this should relieve the problem of this modal dialog sometimes letting other semi-stay_on_top frames obscure it while still prohibiting interaction, wew
  • simplified and sped up how importers decide whether to do file work or wait
  • made file importers do ui-polite pauses during the file import loop more intelligently--if the files make no big change to db or ui (e.g file was known already deleted because of its url), no wait will occur at all, meaning redundant lean queue-sections should work real quick
  • fixed an issue with multiply-nested submenus sometimes not deleting themselves neatly and resulting in highlight-related RuntimeErrors.
  • fixed an issue where the 'select from list' dialog could sometimes be ok'd without a valid entry--I believe this may be have been causing a delayed crash as well
  • added a 'ui timer profile' mode, which profiles some common unified ui update loops that draw animations and keep downloader ui synced and so on
  • the 'delete processed' file import status right-click menu entry will no longer show if it has the same count as the 'successful' line
  • the options->colours panel has some better layout and the namespace listbox asks a yes/no to delete if you double-click/enter a colour
  • dropping a raw file url onto the client will now automatically pend it to a raw url importer (previously, it would be put in the input box, but after rethinking how this could go wrong, I concluded it wasn't such a big deal). if you use this a lot, see how you like it, and let me know if you would want an option to revert to the old behaviour
  • updated domain manager so it tracks url class identity in a neater way (renaming them will no longer break links or display options)
  • improved some widget feedback of edit url class panel
  • simplified the manage url class links panel
  • url classes now normalise gallery and page urls differently (page urls strip all non-declared data, gallery urls just switch scheme). this may change in future
  • url classes can now convert a normalised url to an api url--for instance for informing the program that an imageboard thread url has an api version that'll be better to check
  • the domain manager can now determine arbitrary url parsability capability, and this is just starting to be integrated into url drag-and-drop--try dropping a known url class on the client, like a booru link: it'll now tell you it sees what it is but can't yet parse it!
  • the 4chan and 8chan thread url classes will reset to defaults again this week--replacing with url classes that convert to the api versions--and you'll get the new api url classes themselves
  • if missing, the primary tables in client.caches.db can now regenerate on boot. they aren't properly repopulated yet, but the client will boot and you'll get an error message telling you what to do next
  • improved .txt and clipboard tag import unicode support
  • removed leading BOM (byte order mark) from .txt and clipboard tag imports
  • improved some clipboard text fetching code
  • added a new 'hover window profile mode' to the debug menu--it will print the logic info behind any media viewer's hover window show/hide decision
  • tweaked the locale-setting code, which was stopping some users (on Win10 creator's update, it seems) from booting the client
  • added system:rating to 'all known files' query contexts (so you can now search for deleted files that have ratings)
  • the manage import folders dialog now has a redtext warning about subdirectories
  • some file importers deal with shutdown events more gracefully
  • lz4 is no longer required. if it is absent, or the version is too old, a simple statement will be printed to the log and it will not be used in either the client or server
  • updated to ffmpeg 3.4.1 on windows
  • updated opencv to 3.4 on windows and linux (had trouble with os x--will continue working on it next week)
  • fixed an important grammar typo in the help

• version 289

  • fixed an issue where scrollbars were only appearing on taglists after a resize event
  • fixed the raw filename component of file drag and drop events from the client to external programs
  • fixed the tag lookup scripts
  • fixed some wx menuhighlight issues
  • improved some shutdown code
  • fixed the add/edit namespace colours options panel, which needed to be updated to deal with the new wx's better alpha channel reporting
  • fixed an issue when hitting 'open externally' on a media collection
  • fixed a crash on client shutdown whenever closed pages were in the undo menu
  • think I fixed another shutdown crash
  • fixed a rare issue with the collect by dropdown not being able to generate a string to display
  • misc wx fixes
  • added a 'importing' page to the options dialog, which now sets the default file import options for quiet and loud file import contexts
  • the old and no longer used 'exclude deleted files' option is now removed from the 'files and trash' panel
  • finished off default url matches for all downloaders that come with the program--these will be set on update, so if you have custom ones, please export them before you update so you can import them again!
  • improved how urls are matched and presented for the user in the media viewer
  • added a 'delete "successful" file imports from the queue' entry to the file import status button right-click menu--this only removes 'successful' and 'already in db', leaving anomalies
  • improved locale instantiation in the client and added locale strings to the help->about dialog
  • you can now set the page name prefix for a paused thread checker. it defaults to a unicode pause character: ?
  • thread watchers will no longer pause on a network error during a check--they now have a 'delay' system like subscriptions, and on a network error, they will delay four hours (unless you hit 'check now')
  • patched in some simple 'connection cutoff' network error handling, we'll see how it does
  • wrote in some more proper error handling for a specific connection cutoff error that is being produced
  • the similar files search tree regen code now clears out orphaned files. if you have had blank 'unknown' files appear in similar files searches, please run database->regen->similar files search tree
  • bitmap buttons on download pages will now update using less CPU and will flicker less
  • improved some video rendering error reporting
  • fixed the 'author name' regex favourite default, which had a superfluous asterisk. if you would like to fix it yourself, please try: [^\\]+(?=\s-)
  • added 'flush log' debug command
  • client.pyw now makes a safe 'system' ui error popup if it fails to boot

• version 288

  • updated to wxPython Phoenix (4.0) build!
  • all builds now require the new wx, so if you run from source, please consult the help files for new info on figuring this out
  • did a ton of wx refactoring
  • merged a large number of timers
  • wrote a new wx-aware timer to replace some of the more awkward old timers that could not be neatly merged
  • replaced wx.calllaters (the new ones aren't always garbage collecting nicely) with the new custom timer object
  • misc wx deprecation refactoring
  • cleaned up some wx test code
  • fixed some crashing wx test code
  • fixed 'select from list of strings' object, which was no longer processing enter key properly
  • cleaned up a bunch of ui object interaction code
  • cleaned up a bunch of wx-related shutdown code
  • fixed an issue with the splash window not shutting down cleanly
  • fixed a shutdown db-gui status report crash
  • was unable to get flash embed windows working properly without edge-case crashing on windows, so they are disabled for now (you'll likely get an 'open externally' button instead)
  • animations now clean up their memory buffer faster in certain circumstances--all users who view animations in the preview window should notice significantly leaner 'idle' memory usage over time
  • fixed a very important inefficiency bug that meant select->inbox/archive on large pages with more than a handful of inbox files was taking extremely long periods of 100% CPU
  • wrote a faster way of fetching some media and their paths
  • copying files to clipboard is now much faster
  • initialising any large file drag-and-drop event is now much faster
  • certain redundant image pre-fetching will no longer be done
  • 'open externally' panels will no longer draw a (slightly buggy and useless) animation bar if the underlying media would have had one
  • added a new 'menu profile mode' that will profile any menu click
  • cleaned up the debug menu
  • deleted some old options code that is no longer used

• version 287

  • thumbnails can now be drag-and-dropped to other pages!
  • dragging and dropping thumbs to another page tab will remove them from the source and append them to the destination, maintaining file order!
  • DnDing thumbs to a 'page of pages' tab will put the files in the next lowest selected media page
  • DnDing thumbs to a blank notebook area (or a page of pages without a selected media page) will create a new page for the thumbs
  • holding down ctrl when you drop thumbnails will not remove them from the source
  • please forgive the thumbnail DnD cursor, which for now will be in the 'copy' state, despite the internal DnD being move by default
  • improved page tab drag and drop drop logic--dropping onto the page area itself will no longer send the page to the right-end of the current notebook
  • the 'file import options' object now supports three 'presentation' booleans--for new/already_in_inbox/already_in_archive files--so you can customise whether new thumbnails appear based on each state. page imports will by default show everything, while 'quieter' import queues like import folders and subscriptions will continue to just show only 'new' files in their files popup buttons. if you have a gui page with 10k+ items in its queue, try reducing the presentation to speed it up!
  • all existing import queues will be updated when they are next loaded--but please note that for simplicity and safety they will all initialise to the 'quiet' presentation method, so if you have ongoing download pages in any of your gui sessions (including thread watchers!), they will only add 'new' thumbnails unless you edit them. I apologise for the inconvenience
  • the regular hdd import now has a file import options button!
  • subscription query 'finished' file popups are now merged up to the subscription level--so, a sub with five queries that each get 20 new files in a run will now ultimately leave one popup with 100 files
  • file popups (as produced by subscriptions and a couple other places) now preserve their import order!
  • if a subscription with many queries runs out of bandwidth, it should now only give you one 'no more bandwidth to download files' message, rather than one for every outstanding query to sync
  • added a checkbox to turn on/off the new random subscription syncing to the options->downloading panel
  • the file import status button's menu now supports import/export of sources to clipboard/png! it _should_ also support unicode. be careful not to paste paths into a url cache, or urls from one gallery site to another, or you'll just get errors--this is prototype, so please keep like with like for now
  • the png import/export system now supports raw string payloads
  • the new listctrlpanel can now hang its buttons in multiple rows
  • the manage subscriptions panel now has an 'overwrite checker options' button to mass-set checker options
  • the manage subscriptions panel now has a 'select subs' button that selects subs based on a basic query text search
  • separating merged subscriptions now sets better new subscription names of 'old_sub_name: query_text'
  • saving a session from a page of pages with a custom name will no longer suggest a session name prepended by [USER]
  • doubled the subscription and downloader instance default bandwidth rules to 400 and 200 rqs/day
  • the 'load_images_with_pil' and 'disable_cv_for_gifs' options are now officially BUGFIX in the options--unless you know you need them on, turn them off!
  • added some safeguards to the new dialog-panel system's OK stuff, which sometimes catches a duplicate OK event
  • shuffled some db update status texts around

• version 286

  • simplified how thread watcher assigns DEAD and 404 status--it should do it more reliably now
  • thread watchers now publish their page names (including updating dead/404 status) right after they check
  • the save session menu under pages now lists the exsting sessions for quick-save--if you select one of these, it will throw up a yes/no overwrite confirmation dialog
  • 'appending' sessions now loads them in a new page of pages named as the session name!
  • right-clicking on a page of pages now gives you a 'save this page of pages to a session' option
  • right-clicking on the page tabs lets you append a session anywhere
  • 'load' session is now 'clear and load', and it throws a yes/no dialog for confirmation
  • the gallery pending queries box now allows multiple selection with ctrl/shift clicks. it'll even move up/down correctly on non-contiguous selections!
  • the gallery query input is now a new combined textctrl-and-paste button control that will appear in other places across the program
  • the page of images input now also uses this new control
  • the page of images and raw urls queues will now ignore any input that does not start with 'http'
  • pasting a list of texts from the clipboard will now typically strip the pasted content of leading or trailing whitespace
  • some text and tag cleaning is a bit faster and neater
  • the file import status panel now supports 'open sources' on its right-click menu--it opens in your web browser or file explorer as appropriate. (linux can't do file explorer though)
  • thumbnails will no longer 'fade' when their pages are not visible, reducing CPU load in several contexts
  • thumbnail fade will use less CPU overhead on very fast computers
  • the timer loops that all import pages use to keep themselves updated are now harmonised into one loop at the top gui level (this should reduce some idle CPU overhead and improve some ui responsiveness)
  • content (tag and rating) shortcuts can now be set to 'flip on/off' or 'set'
  • the manage subscriptions dialog now waits for any currently syncing subs to cancel and save themselves before opening itself (keeping it all synced and rollback-proof)
  • added a 'watchable' url class type and wrote examples for 4chan/8chan thread urls
  • added 4chan/8chan file url classes to the examples, which if added will auto-hide them from the media viewer
  • fixed 'add parameter' in edit url match panel
  • simplified some stuff in the existing parsing system
  • added preliminary 'url type' support to the parsing system
  • the html formula parsing object can now return the full child html of the node it finds
  • the new url and html stuff doesn't do anything yet--but it will in the new downloader engine
  • added a bunch more booru entries to the url classes defaults. I will continue filling these out, hopefully setting comprehensive defaults next week
  • the advanced mode thumbnail right-click menu lets you reparse and regen thumbnails for any files! if you have rotated or stretched files, you can now fix them from the menu!
  • added new tools to let the client update media file metadata and thumbnails live!
  • the new reparse and regen should update file metadata and thumbnails live!
  • improved how some 'quiet' errors are printed to the log. they'll now have the full trace of the error as well as the stack
  • added a daemon_report_mode debug mode. it throws up a popup every time a daemon fires its callable
  • fixed an issue with the editstringtostringdictcontrol
  • fixed an issue with loading (and updating to a new version) import pages or import folders with some kinds of unicode paths
  • you can now set the 'woah, you should close some pages' warning value under options->gui. its default remains 165 pages
  • system:similar_to searches should be a lot faster on clients with many files
  • the manage subscriptions and edit subscription panels now both list their num_urls summary column as '22' if all done but '11/22' if not. this column also sorts based on percentage completion, then num total, then num done
  • the database migration dialog now uses the new listctrl--its buttons are now also clever, and will disable when they are invalid
  • misc fixes
  • fixed an index bug in the new listctrl after certain types of 'setdata' call
  • cleaned up some search code
  • some veto code cleanup
  • cleaned and harmonised how text is pulled from the clipboard

• version 285

  • added 'network' main gui menu and moved network stuff from services to it
  • split the new domain manager stuff up into separate dialogs and menu entries on the new network menu
  • manage url classes dialog now lists url type in the listctrl
  • added url class links info (which will permit client-specific settings and downloader mappings for url classes) to the domain manager
  • wrote a 'url class links' dialog and added it to the new network menu (only the 'display on media viewer' part works atm)
  • the domain manager now filters urls on the media viewer depending on whether they have a url match and are set to display in the new links panel
  • updated the local booru service code to the new service system
  • the local booru's shares can be reviewed again under review services
  • the local booru's port and bandwidth settings can be set again under manage services
  • the different gui parts of the local booru are updated to new controls
  • fixed a local booru 404 reporting error
  • the edit subscription panel now has a 'paste queries' button that lets you add queries en masse
  • added 'manage_file_urls' to shortcuts system
  • added several 'get_similar_to_x' actions to the shortcuts system
  • the manage upnp dialog now initialises its mappings on another thread and fails better when UpnP mappings cannot be fetched
  • connection and readtimeout network exceptions are now recognised more specifically by the client. subscriptions will only delay for an hour on one of these exceptions
  • improved the resilience of the HF login code after wake from sleep (when networking is often not available for a couple of seconds)
  • like the recent subscription query randomisation update, subscriptions themselves are now synced in random order (this stops a subscription named 'aadvark' always getting first bite into available bandwidth)
  • fixed import for jpegs that have unparsable exif data
  • fixed a bug in 'next month' bandwidth estimate calculation when the month is December, wew
  • fixed some logic that was setting max page at 165 rather than the intended 200 and added a dialog that even lets you engage the debug override at 200 if you are insane
  • all audio mime detection and duration parsing is now done through ffmpeg. hsaudiotag is no longer needed to run the program
  • since the old listctrl sort crash bug is still hitting some people, I've disabled sort on the old listctrl class. feel free to try to sort any listctrls now, no matter your situation. I will continue replacing the old class with the new working class over time
  • updated another listctrl
  • a ton of misc controller options/manager access refactoring
  • cleared out some old code
  • moved Time controls to their own file and added velocity and checker options stuff as well
  • wrote a new edit panel for single controls and updated the time delta button to use it
  • misc refactoring

• version 284

  • fixed subscription queries turning dead on the initial sync
  • all dead subscription queries have been set to check again in case they can revive
  • added query file velocity to edit subscription panel
  • subscription network contexts now reflect the new multiple subscription query system, and are named "sub_name: query_text". as every query now counts as its own separate subscription network context, this will stop query-heavy subscriptions from throttling so much on bandwidth limits
  • finished URLMatch object, which matches and normalises URLs into certain 'classes' like 'gelbooru post url'
  • expanded some URLMatch subdomain options
  • fixed some test logic in URLMatch
  • finished the last of the EditURLMatchPanel
  • split the 'manage network rules' dialog into two panels--it now has a 'url classes' tab
  • wrote a panel for managing url matches
  • added export/import/duplicate buttons to EditURLMatchesPanel
  • wrote some URLMatches for hentai-foundry as an initial test of the system and added a temp button to add them--please check them out to see how it all works
  • all, invert, inbox, and archive (and none, lol) thumbnail 'select' menu items now have counts
  • invert is now at the bottom
  • the thumbnail select menu now has local/remote entries if applicable (this typically is only true in 'all known files' file domain)
  • added png/clipboard export/import/duplicate code to the generic new listctrl button wrapper panel, which will save a bunch of time as the png/clipboard sharing system expands
  • added a human-facing serialisable name to all objects on the new serialisation system and tied the new import/export code into it for png presentation
  • the edit import folder dialog will now complain (but not veto) on an ok event if any of the entered paths do not exist
  • if you attempt to manually run an import folder while import folders are globally paused, you'll get a little popup telling you so
  • added an experimental 'thumbnail fill' setting to options->gui. it zooms the existing thumbnails so they fill the whole thumb space. feedback on this from those who would be interested in a prettier system would be appreciated
  • added 'paste tags' buttons to filename tagging options panel
  • the paths/urls in the file import cache are now their own object that holds the creation/modified/source times and current status and note. this object can also hold prospective urls, tags, and hashes for future use
  • a bunch of file import actions are faster
  • all the different importers now use this new file import object
  • fixed a screen position calculation in the new drag and drop filtering code that was accidentally including too many possible drop candidates on drops in the top-left corner of the main gui (if you had trouble moving tabs to the left, this should be it fixed!)
  • fixed a problem display volume/chapter/page tags that included unicode characters in thumbnail banners and media viewers
  • fixed a rare media display bug in the dupe filter
  • fixed some 'C++ part of panel has been deleted' bugs in review services if the frame is shut down before delayed db info is fetched
  • cleaned up some more 'C++ deleted' errors in import files selection dialog
  • fixed the network context custom header panel 'add' action, which wasn't saving the value of the panel
  • fixed a bunch of bugs in the newish QueueListBox class
  • SynchroniseRepositories daemon will be better about quitting early on application shutdown
  • cleaned up some pending pretty timestamp grammar
  • when the client cannot clean up a temporary file, it will print more error information
  • added pylzma to the 'running from source' library recommendations. this is not required, but if available it adds ZWS flash support

• version 283

  • subscription popups show a bit more info about their individual queries
  • subscription popups hide the sub-query info in text and file buttons if it is the same as the subscription name
  • subscriptions now work through their queries' downloading phases in random order (so a heavy query named 'aardvark' won't repeatedly choke the sub's other queries if bandwidth is tight)
  • if a subscription query dies, a notification popup will now be made
  • changed a logical edge case that may have been killing some subscription queries after the first sync
  • re-enabled pixiv artist search, as it still uses the old gallery style
  • added tentative support for 'ZWS' flash files
  • if there is more than one import folder, the 'check import folder now' submenu now has a 'check all' entry
  • TimeDelta controls that have the 'days' entry (as in thread/subscription checker options panels) now have a max period of 10 years for those _really_ slow threads
  • fixed an issue where the 'quick namespaces' section of the new filename tagging options panel was not initialising correctly when it loaded data from a previously serialised import folder (this was the "unhashable type: 'list'" bug in the edit import folder panel)
  • the filename tagging dialog now works on the new panel system. it also has an entry under options->gui: 'local_import_filename_tagging'
  • polished StringMatch object
  • polished EditStringMatchPanel
  • html formulas now have a string match object
  • improved parsing error system
  • miscellaneous improvements to the parsing system
  • misc fixes to StringMatch and parsing system
  • the main gui will no longer interpret text drops on child frames as url drops. this fix just quashes the drop--it doesn't allow the initially desired drop on the child frame to occur. I will keep thinking about this
  • the dupe filter's 'no media to display' text now says 'Looking for more pairs--please wait'
  • the 'remove' media action is now labelled 'remove from view' to reduce 'is it a delete?' confusion
  • fixed an icon initialisation order problem that meant pre-boot errors were unable to display
  • fixed a graceful-shutdown-on-boot-error bug
  • the custom 'listboxtags' controls now post an event on their contents changing
  • the tag censorship panel now updates its status text when its listboxes change due to a mouse double-click event
  • fixed file import 'source time' column sometimes showing '3e-05 years' kind of garbage that was due to a missed float->int conversion
  • cleaned up how some page names are thrown around, which should stop some stray [USER] prefixes slipping though
  • the adminside petition management checklistbox will now escape mnemonic characters (typically '&') and hence display them correctly on sibling pairs and so on
  • wrote a 'help my client will not boot.txt' that explains the new debug builds and some other common fixes to try
  • 'all known files' file search domain is now hidden behind advanced mode
  • fixed another instance of flashwin causing a failed boot
  • cleaned up some old broken unit tests
  • some misc help improvements
  • updated Windows build to new version of sqlite

• version 282

  • rolled back to an older version of pyinstaller that does not seem to have the embedded flash window problems windows users experienced
  • added an error handler to wx.lib.flashwin import--if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window
  • created a 'filename tagging options' object to handle the instance-non-specific auto-tagging parts of the filename tagging dialog
  • moved all the appropriate tag generation code to this new object
  • extracted the simple and advanced panels in the filename tagging dialog to a separate panel
  • wrote a new wrapper panel to edit filename tagging options objects directly
  • cleaned up some tag generation code--it'll now all be siblinged/parented/censored the same way
  • import folders now use the filename tagging options object, harmonising this tag parsing code
  • edit import folder gui now has an add/edit/delete listctrl to manage the new tag_service->filename_tagging_object relationship (this replaces the old .txt management button and summary text)
  • finished the StringConverter class in the new parsing engine. it applies a series of transformations to text
  • wrote a panel to edit string converters
  • wrote a panel to edit string converters' individual transformations
  • updated html formulas to use string converters instead of the old cull_and_add system
  • html formula edit panels can now edit their new string converters
  • file lookup scripts now 'stringconvert' their file identifier strings--this should allow the 'md5:md5_hash' fix we've been talking about (i.e. by first encoding to hex and then prepending 'md5:')
  • the help->about window now shows the client's default temp directory
  • thumbnail regeneration--particularly full-size regen and resized png gen--should be much faster in the client
  • the debug exes are now included in the windows build
  • the debug exes are now included in the non-windows builds
  • the test exe is no longer included in the windows install (can't remember it ever being useful, and it is like 10MB now)
  • unfortunately, a recentish change to how pixiv serves gallery page results has broken the hydrus pixiv downloader. this will have to wait for the downloader overhaul, and even then, it might be _slightly_ tricky. pixiv downloader entries are now hidden, and existing pixiv subscriptions will be paused on update
  • the thumbnail select submenu now clarifies 'all (all in x)' if all are in the inbox or archive
  • you can now archive/delete filter from the thumbnail menu on a single file
  • lowered the http session inactivity timeout even more, to 45 minutes
  • fixed a couple of instances of the new subscription dialogs being unable to mass-retry failures
  • ffmpeg parsing errors now print more info about broken file path
  • some daemons will be snappier about shutting down on application shutdown
  • took out the sometimes invalid 'booting' phrase in the during disk cache population status report
  • the client will now warn you about page instability at 165 pages after one user reported instability at 175. 200 is still the strict limit.
  • downloader pages and subscriptions will fail more gracefully if their downloaders cannot be loaded (e.g. they used a since-deleted booru)
  • fixed listctrl panel button enabled/disabled status when the child listctrl starts empty
  • the new listctrl can now select data in a cleverer way
  • misc fixes
  • misc refactoring

• version 281

  • subscriptions can now support multiple queries!
  • subscriptions now use the new thread checker system to set their check periods!
  • improved some check timing calculations to deal with subscription timestamps
  • existing subscriptions will update
  • subscriptions can now see any 'delay' reason on their edit page and can scrub it there as well
  • subscriptions no longer track 'last error'. everything goes through the delay system and keeps track of reason
  • added 'last new file time' to the subscription listctrls
  • subscriptions now generate separate 'file button' popups for each query, and file button popups now include their name in the button label
  • the subscription listctrl is now breddy wide. I will be adding some way of hiding columns (with persistent memory) on the new listctrl at some point in the future, so please bear with it for now
  • manage subs now has a wrapper panel and buttons that disable when invalid
  • tumblr simple 'video' posts should now be supported! turns out they are just mp4s
  • in the client, static image thumbnails are now generated with opencv
  • files that give truncated image errors should now import ok. they'll render with errors. if cv is up to date and working ok, they should have good thumbnails as well (otherwise they will have all-black thumbs)
  • updated manage regex favourites dialog to the new panel system
  • regex favourites are now editable under the new options->regex page
  • ongoing import or export folders will now stop working if their respective manage dialogs are opened
  • import folders can now stop working if the client shuts down during the initial file parse phase
  • export folders will now stop working immediately if the client shuts down
  • export folders can now stop working during the file copy stage
  • the splash screen is now a little wider and taller
  • the splash screen now has a subtext row for misc technical data
  • the boot and exit splash screens show a bit more data about several things
  • the repository sync daemons should be a bit snappier about responding to a client shutdown
  • the import files dialog is now a non-modal frame! (you can now continue to use the client while this frame parses files) you can even open up several at once!
  • the 'import file path tagging' dialog should now never be taller than its spawning screen and also slide up and left if it spawns with its bottom-right corner off-screen (this dialog is still on the old system, however, so the newer sizing stuff that does this stuff automatically is not yet available)
  • sliding up-and-left should now also occur on spawn for frames on the new system
  • the advanced 'process now' repository button will now disable itself if there is no work to do!
  • if simple downloads like gallery pages are larger than 100MB, the client will throw an error (this is a stopgap for an ongoing issue)
  • the 'maintenance and processing' page has a bit more warning text
  • the new listctrl wrapper panel now supports menu buttons
  • libpng's iccp terminal warnings from loading the flash/pdf/zip thumbnails should now be gone!
  • misc refactoring

• version 280

  • the client should now recognise EXIF file rotation and flipping in resolution discovery, thumbnail generation, and image rendering. existing files will be borked, so will have to either be manually deleted&reimported, or wait for a maintenance routine that will retroactively fix all these
  • videos with a non 1:1 sample aspect ratio (wew lad) should now import with correct resolutions
  • all 'tag import options' are now managed with the new button
  • deleted all old tag option management gui, including deleting ClientGUICollapsible entirely
  • with the simpler tag import options, increased max page count to 200
  • page of images downloader now has two pause separate buttons for queue and files processing
  • the page of images downloader should no longer too-quickly blank out its parser status at the end of queue processing
  • the default hydrus bandwidth rules now no longer has the choking 'requests per day' limitation that was messing up large mappings uploads. (it is now just 64MB/day)
  • your hydrus default bandwidth rules will reset to this on update
  • POST requests (which in for our purposes are always user-driven) no longer obey bandwidth rules
  • fixed a bug when sometimes hitting the file limit or hitting a siteside 404 in the gallery downloader
  • a rare bug due to empty string menu labels is now no longer possible. empty string labels, if they slip through, will be replaced with 'invalid label'
  • you can no longer choose the empty string as a gui session name
  • improved how network jobs keep track of all applicable network contexts vs the preferred session and login contexts
  • cleaned up a bunch of login code, improved verification testing
  • fleshed out domain login system
  • in the networking engine, the read timeout is now six times the connect timeout--we'll see if this reduces some hydrus and other read timeout problems we've seen
  • increased the server transaction period from 10s to 120s--we'll see if this helps reduce some spiky server POST lag or what
  • the default tag import options panel now edits on double-click, rather than deletes. it also yes/no dialogs on a delete action. this control is still trash, but it now works more like the rest of the program

• version 279

  • moved the hydrus service object over to the new networking engine
  • hydrus services now appear in the bandwidth review panel
  • hydrus requests now obey larger network bandwidth rules (this mostly means 'global')
  • service bandwidth usage and rules are no longer managed from manage services--it is now under review bandwidth usage, like all other network contexts
  • updated some network engine stuff for misc hydrus states like 'server busy'
  • fixed a bug in the server code where the session key cookie had an invalid expiration timestamp--servers should update this week to make sure new clients can log in properly
  • hydrus network requests will force-set User-Agent as 'hydrus client/(network_version)' as the network version is used in the protocol to determine compatibility
  • fleshed out the hydrus specific network job, giving it the various bandwidth tracking and version checking responsibilities the service object used to have
  • moved session cookie decay (only matters for Hentai Foundry atm) to the new session manager (was previously hacked into the login manager)
  • moved some hydrus response parsing stuff around, added content type awareness to the new network job
  • updated several unusual hydrus 'static/test' requests used to test credentials and fetch access key and so on to the new system
  • added a special 'test service' service to better accomodate these requests
  • wrote a static login script for hydrus services
  • polished up login management system overall
  • import folders now support a 'check now' state, like subscriptions, that will cause them to check immediately
  • import folders can be 'check now'ed from the file menu under a new submenu! if you would like to have a 'manual' import folder, try pausing it and just running it from this menu!
  • added an optimisation to the file search algorithm to search ratings queries super fast when they lack tag or file system preds to otherwise speed them up
  • updated the booru presets that now support https to be https
  • added a 'restore defaults' button to the manage boorus dialog--you can restore specifics or all of them
  • optimised how fading thumbnails are blitted to screen, which may provide a huge performance boost for high-res/small-thumb clients
  • pages that have been renamed by the user will no longer be rename-overwritten by any auto-renaming system (currently just thread watchers, I think, but this will expand in future). unfortunately, this is not retroactive--only pages renamed from now on will be aware that they were user-renamed
  • in advanced mode, the pages menu now states how many pages are currently open
  • the 'page of images' downloader will now say '(x already in queue)' when it reports how many urls were found, if any were already in the queue. (this should clear up some confusion where it would previously say '0 new urls' even when it found some stuff)
  • the gallery downloader will do essentially the same, but on a per-page basis. the text is a little crushed, so I may revisit this
  • fixed an issue with the manage services dialog not being able to rename dupe-named services on edit subdialog ok
  • the manage services dialog now uses the new listctrl! the listed services are no longer a horrific unsorted mess!
  • fixed the file import status button not showing on raw url downloader or import folder edit dialog
  • improved how some directory tree merging code deals with read_only files in the source
  • maybe fixed some unusual selection behaviour with the booru selection popup dialog. it now has a real ok button, rather than a mickey-mouse hidden one
  • completely deleted the old networking engine!
  • cleaned out some unused imports from the networking code and related entries from running_from_source.html
  • changed up and fixed some odd bugs in how how repositories test some error/isfunctional stuff vs regular paused status
  • hydrus network contexts now have a prettier name
  • ip address-based network contexts will no longer spam the bandwidth tracker with their useless subdomains
  • network jobs that are unable to attempt validation or login will now error immediately rather than waiting indefinitely
  • fixed up a bunch of test code that was broken due to the mismatch of network engines
  • broke some other test code due to the network engine transfer!
  • the client is more resilient about broken 'twisted' installs, and _should_ be able to boot without it. this may or may not apply to the built release--more work can be done here
  • some network job refactoring
  • clientside service code cleanup
  • misc fix that I can't remember
  • misc cleanup
  • misc refactoring

• version 278

  • fixed the tumblr raw url converter to now point to the data.tumblr.com domain
  • added a hardcoded ssl verify exception for data.tumblr.com, which has an incorrectly defined ssl cert (at least for public-facing interactions), wew
  • all existing db urls and file import cache urls for media.tumblr.com will be updated to data.tumblr.com on update! (everything _should_ just magically work again)
  • fixed apng import, which the decompressionbomb detection code was not handling correctly
  • collapsed the different instatiations of the 'file import status' button down to one class
  • the file import status button now has a right-click menu that supports 'retry failures' and 'delete processed', if applicable
  • misc import status cache cleanup and refactoring
  • you can now edit or completely turn off the [404] and [DEAD] thread watcher page name prefixes under options->downloading
  • thread watchers should more reliably keep 404 status
  • 'open selection in a new page' now preserves file order!
  • 'view this file's duplicates' now sorts the files!
  • options->gui now has an option to change how often 'last session' is saved
  • 'last session' will no longer autosave to the database if there are no changes
  • tags exported to neighbouring .txt files are now correctly sibling-collapsed
  • tags imported or exported via neighbouring .txt files are now correctly tag censored
  • the manage tags dialog will now protest with a yes/no dialog if you attempt to cancel it with uncommitted changes
  • the manage parents and siblings dialogs will now protest with a yes/no on an ok event if there are 'uncommitted' pairs in the lower boxes (e.g. if you forgot to click the 'add' button)
  • fixed an issue that would sometimes stop old sessions from loading properly
  • the duplicates page now does its maintenance jobs in modal popups!
  • attempting to apply a duplicate status to more than 100 pairs now throws up a warning yes/no dialog
  • the manage urls dialog now has copy/paste buttons
  • added a (somewhat debug) option to disable the mouse hide&anchor behaviour on slow Windows canvas drags to options->media
  • added a 'regen all phashes' command to the database regen menu
  • the disk cache options in help now have a help button to explain good values for ssd vs hdd
  • the edit bandwidth rules dialog now uses the new listctrl
  • merged the old and new login managers
  • misc login work
  • misc refactoring
  • misc cleanup

• version 277

  • expanded the domain manager into a legit serialisable object that holds data and saves itself on changes. to begin with, it supports custom http headers for particular on network contexts
  • the global User-Agent (which now defaults to 'hydrus client') is set through this new manager, as attached to the 'global' network context
  • wrote a basic panel to edit custom http headers under services->manage network rules (this panel will be fleshed out with more in future--please ignore if you are not an advanced user)
  • wrote a panel to edit an individual custom http header
  • wrote a panel to edit/select network contexts. this will get a bunch more use elsewhere as the overhaul continues
  • flushed out and tested the new popup yes/no button system
  • sanakucomplex.com has a User-Agent entry in the new custom header system--on the first sank request, you will be presented with a yes/no popup asking if it is ok to use.
  • if you ok the user-agent, sankaku now seems to work again!
  • sankakucomplex.com network context will get some specific conservative bandwidth rules (80 rqs per 7m, 1 rq per 4s) on update
  • added an option to options->gui to reverse the page tab shift behaviour
  • 'don't scroll down on key navigation event if thumb is at least this % visible' value is now 75%. it is also editable under options->gui
  • the hydrus icon used on frames is now the non-transparent version that shows up better on dark backgrounds
  • the standard hydrus.ico used in web favicons and the executable builds is now also non-transparent (this should also propagate up to OS shortcuts and taskbar icons wherever the frame icon is not inherited)
  • file export drag and drop events will now defocus the currently focused media on successful drop (e.g. if you drag a video to an external media player, it will stop rendering clientside), just like for open externally
  • database analyze maintenance should be more reliable with respect to fresh repository syncs
  • reduced default impermanent session timeout in new networking engine to 60m (should fix/reduce some hentai foundry 503s)
  • misc ui improvements and speedup
  • misc fixes

• version 276

  • the new thread watcher object will no longer produce check periods shorter than the time since the latest file. this effectively throttles checking on threads that were posting very fast but have since suddenly stopped completely
  • thread watchers now parse their thread subject and place this in the left management panel
  • thread watchers now name their pages based on the thread subject, if one exists
  • an option to permit or deny thread watchers renaming their pages is now under options->downloading
  • dead and 404 threads now disable their checker pause button--to attempt to revive, hit 'check now'
  • thread watchers now preface their page name with [DEAD] or [404] when appropriate
  • misc thread watcher code improvements
  • added basic import support for zip, rar, and 7z files. they get no useful metadata (yet) and have a default 'archive' thumbnail
  • the client will now by default detect and not import decompression bombs before they blat your computer. an option to allow them nonetheless is under options->media
  • the server will now not parse or accept decompression bomb uploads in POST requests
  • added a 'refresh all pages' entry to page of pages's right-click menu
  • added 'send this page down to a new page of pages' to page right-click menu
  • added 'send all pages to the right to a new page of pages' to page right-click menu
  • fixed a page of pages drag and drop issue when dropping the last page of a notebook onto the same notebook tab
  • fixed some index calculation problems when DnDing page tabs to the right on the same notebook
  • sending a refresh event to a 'show selection in a new page' page (which has no search predicates and so cannot 'refresh' its search) will now trigger a sort event (like importers got last week)
  • thumbnails at the bottom of the current view but are at least 90% in view will no longer scroll into view when selected
  • click events will no longer scroll thumbnails that are semi-out of view into view
  • improved how all 'wait until the client ain't so busy' checks work. importers that have a whole slew of 'already in db' to catch up on should now not clog the gui so much
  • similarly, under ideal conditions where nothing is busy, importers will iterate over their files more quickly
  • the network engine now has a 'verification' loop that doesn't do anything yet, and a stub domain engine is generated to be consulted in this
  • wrote some verification code, extended popup messages to support yes/no questions
  • polished some domain engine code
  • fixed an issue where file repositories were not recording deleted files in certain cases
  • all file repositories will be reset on update
  • the date entries on the review bandwidth bar chart now have leading zeroes on 0-9 months to ensure the sort correctly (this month's 2017-10 entry was sorting before 2017-8, wew!)
  • the migrate database dialog now shows approximate total thumbnail size
  • gave the migrate database help a quick pass
  • gave the 'help my db is broke.txt' file a quick pass

• version 275

  • if you hold shift down while dropping a page tab, the client will not 'chase' that page to show it (try it out!)
  • the gui will be more snappy about dealing with drag-and-drop drops (of types file import, page tab rearrange, and url drop), returning the mouse to normal state instantly on drop and dealing with the event in a subsequent action
  • dropping url text on the client will specifically inform the source program not to delete the dragged text (that the operation was copy, not move), if it is interested in hearing that
  • page drag-and-drops should transition a little less flickerily
  • all file import status objects can now track 'source time', typically to represent upload time
  • file imports now populate 'source time' based on the earliest of creation/modified time! (this will be used later to parse as a tag)
  • thread watchers now populate 'source time' based on post time!
  • finished a watcher options object for the new thread checker system
  • wrote a panel to edit watcher options
  • converted the thread object to the new watcher system
  • thread watchers now have two pause buttons--one for the file queue, one for the checker
  • compressed thread watcher ui layout
  • converted the thread left-panel ui and options->downloading page to reflect the new watcher system
  • improved the watcher options to generate better timings for fresh threads
  • cleaned up some thread watcher check time code
  • the total/selected mime summary on the status bar is a little prettier and will now report by individual mime sometimes
  • generating the total/select mime summaries are now faster on pages with >1,000 files (it'll just use 'file')
  • a 'refresh' action on an import page now triggers a sort event
  • added 'flip_darkmode' shortcut command to 'main_gui' and 'media_viewer' shortcut sets
  • added copy_file/path/hash/bmp actions to the 'media' shortcut set, removed the hardcoded ctrl+c for copy_file. I added ctrl+c to the defaults, but existing users will have to re-add it manually if they want it!
  • simplified some page ui update code
  • import pages will no longer update their left-panel ui (which uses a bit of cpu) when they are not in view
  • polished some new string and url matching code the domain engine will be using
  • wrote a panel to edit string match objects
  • wrote a new panel to handle simple ordered lists of data in a better way
  • wrote most of a panel to edit url match objects
  • misc domain manager work
  • fixed an issue with the old listctrl where object name de-duplication was sometimes not permitting (1)-type names to be cleaned up
  • fixed some inelegant time duration->text conversions
  • improved complete process shutdown reliability when some downloads are waiting on bandwidth on shutdown
  • did some misc listctrl update work

• version 274

  • the help menu now has an easy on/off check entry for the darkmode colourset
  • changing any individual colour or the entire colourset will now immediately refresh almost all custom-coloured controls with the new colour
  • added a BUGFIX option to options->gui to permanently fix all discord file drag-and-drop events (as long as they contain <= 10 files and total < 50MB)
  • you can now set specific 'open externally' launch paths on a per-mime basis under options->files and trash
  • improved error reporting on a bad file launch
  • improved the network engine to recover from and reattempt in-progress response read errors (previously reported as ReadTimeout)
  • fixed the 'scroll to focused thumbnail' calculations on key events (when hitting up/down arrow key on thumbnail grid, the page wasn't scrolling correctly as needed)
  • known urls no longer display with the scheme (http or https) in the media viewer top-right summary
  • known urls in the media viewer top-right summary now tooltip their full url
  • wrote a new button for editing tag import options. it has a good summary tooltip. it is only in the manage import folder dialog atm, but I will replicate it across the rest of the program in the coming weeks
  • in some situations, the file import status window will list some timestamp note info for 'already in db' and 'deleted' statuses. see if you like it
  • the new listctrl will generally give its data to other consumers in ui sorted order (this fixes some stuff like 'copy sources' in the file import status window, which was copying them in random order)
  • manage tag parents now uses the new listctrl (and is hence now safely sortable)
  • manage tag sibings now uses the new listctrl (and is hence now safely sortable)
  • some behind-the-scenes of manage parents/siblings is a bit neater
  • improved some thumbnail internal media structures to fetch specific media based on hash much much faster, particularly for pages with 10,000+ thumbs. this should speed up large imports and other content update events that can result in thumb redraws
  • fixed an issue where dismissing a popup message could spawn the entire result of the queue, ignoring the 'show 10 max' rule
  • completely finished the menu rewrite! all menus now work on the new system
  • deleted a ton of old and now obselete menu event processing code
  • some boot/shutdown terminal printing should be more reliable
  • misc refactoring
  • misc cleanup

• version 273

  • converted the gui colour options to the new options system
  • added a 'darkmode' colour set and extended the colour options page to support it!
  • added a secret discord-compatible drag and drop mode. start a file transfer drag and drop with ctrl held down to initiate. keep ctrl held down, as this is secretly a 'move' drag and drop, and you want to keep it a 'copy' one. it should work for discord dnd, at least to the web version. I have an idea on how to do this better, but feedback on this would be appreciated.
  • converted 'import options - files' collapsible panel to a button that launches a dialog. this reduces total gui object count when many importers are open and reduces overall ui lag and limits. the button shows a summary of its current options as its tooltip. furthermore, all of this sometimes confusing nomenclature is now uniformly presented as 'file import options'
  • added a BUGFIX option to gui options page to force hover windows to display at all times
  • improved last week's gelbooru 404 fix--page urls generated pre-v272 should now be more reliable
  • in a further attempt to improve workspace support, popup messages will no longer update in any way if the mouse is not on the same display as the main gui
  • fixed an issue where dupe import urls could be queued up in a file import queue if they came in the same batch. this was essentially harmless but lead to some mixed x/y progress counts and row indices
  • fixed an issue where url drops were not filling in the url entry when a new raw url page had to be created on a currently selected page of pages. this may have affected a couple of other page spawning situations
  • may have fixed an issue with the mouse shortcut-setting button, where for some users scroll events were not registering
  • manage subs dialog now has the new help button up top
  • manage subs dialog's sub verbs are now wrapping into a single menu button that I can expand more easily in future
  • sped up thumbnail access a bit on certain well-defragged hard disk drives
  • made thumbnail fading a little smoother in some situations
  • rendered images that aren't otherwise pushing any memory limits will now stay cached for less time (10 mins) while thumbnails will hang around for longer (24 hours)--this was previously 20 mins for both. you should now see less thumbnail page 'refreshing' after returning to the client after inactivity
  • sketched out basic classes for login and domain engines
  • sketched out basic class for url matching
  • sketched out basic classes for matching and transforming strings in certain login/domain situations
  • updated the manage export folders dialog's listctrl to the new control
  • updated the last media viewer menu code to the new system
  • removed some old proxy code that was sometimes intruding on the new networking engine. proper proxy support will come in a later version
  • fixed some test code
  • misc refactoring

• version 272

  • finished moving the last misc network consumers to the new networking engine
  • greatly simplified and harmonised some of the new network job response processing
  • the gelbooru 404 issue should be fixed!
  • added a parsing patch for rule34hentai.net's mp4 links, which was causing 'link not found' failures on the hydrus parsing end. please tell your R34 subs to retry their fails
  • wrote a patch (actually updated an existing patch that had fallen to bitrot) for the danbooru booru (and extended it to anything else with 'Running Danbooru' in its html) to try to fetch the full-size image if one exists. this may not last long, so I do not recommend regular users rely on it, but feel free to play with it until I have a better fix in the new downloader engine
  • you can now add 'thread watcher' to the default import tag options. it has 'filename' and explicit tags options. unlike gallery downloaders, thread watchers will not inherit from the 'default' import tag options entry--you need to set the specific entry
  • added regular number key support to the new shortcut system
  • fixed the borked splash screen drag coordinate calculation
  • heavy session loads should be more polite to CPUs and window managers
  • pages should generally spawn with their proper name (rather than a flicker of 'page') no matter their spawning context
  • fixed osx page tab drag-and-drop coordinate calculations
  • figured a way to make os x page tab drag-and-drop start on a single left-mouse-down event
  • in an effort to improve popup display on systems with virtual desktops, the popup message parent window will now not go from hidden to shown while the mouse is on a display screen other than the main gui
  • fixed a wacky bug where clicking on a page tab, then on its sort dropdown, then on a different tab, would invoke the Deep Ones to commence a page tab drag and drop event
  • fixed an index issue when unclosing some pages in certain orders
  • increased pre-processing disk cache population time
  • tweaked regular memory maintenance disk cache population timings
  • fixed shortcut-driven tag petition events when the tag has a sibling (e.g. if you have sibling pair (a->b) and a shortcut (key->flip a), hitting 'key' will now correctly flip a (appearing as b) on and off in the media viewer)
  • fixed some ugly layout sizing calculations on the path tagging dialog
  • updated the path tagging listctrl to the new control
  • updated the quick namespaces listctrl to the new control
  • updated the regexbutton's menu to the new menu system
  • fixed some typo errors with the regexbutton's entries!
  • removed the old gallery downloader unit tests, as these operated on the old network and downloader engine--new, granular tests will be reintroduced for the new parsing components as they are written
  • cleaned up some shortcut key code
  • misc refactoring
  • misc improvements

• version 271

  • fixed an issue where the notebook 'motion' event was being consumed, disallowing the regular OS 'highlighting' on tab mouseover
  • fixed page tab drag and drop crash for Linux
  • fixed page tab drag and drop coordinate calculation for OS X
  • page tab drag and drop should now work for all platforms
  • new 'page of pages' pages now start with a blank file search page
  • fixed several issues with row display in the new listctrl
  • added a # index column to the file import status window, so you can resort the 'to be imported' order reliably
  • fixed an issue where the 'delete_file' application command was not firing on an appropriately mapped shortcut
  • fixed an issue where automatic bandwidth override would only fire if its page was in view. it now occurs at all times
  • fixed an issue where the file import status frame, as launched from manage subscriptions, would sometimes be non-responsive (typically for Linux)
  • fixed the 'pattern shortcuts' button on the export files dialog, which was misbehaving for certain choices for some window managers
  • fixed an unusual error where middle-clicking to create a new page while a menu was open would lock the application from receiving new key/mouse events until program focus was lost, wew lad
  • updated some old network code to the new engine
  • added some delay to network connection error reattempts so that instant fails due to local network disconnect (e.g. while computer is waking from sleep) do not consume all the reattempts too quickly
  • greatly simplified low-time-delta bandwidth testing and reporting, improving essential accuracy and overall feel
  • gallery download errors will now print additional info to the log--let's see if we can gather more info on the gelbooru 404s
  • wrote a patch for users of 2.X opencv who were running from source, who were unable to boot
  • due to hardware failure, had to make a new Linux build computer. some libraries are updated, but build _should_ be the same--let me know if not
  • updated to opencv 3.3 for all releases
  • misc old menu code updates

• version 270

  • added page tab drag and drop (windows only, have to iron out critical bugs for linux/os x).
  • dragging onto the middle of a normal tab will put the source tab there
  • dragging onto the edge of a tab will try to insert the tab there
  • dragging onto the middle of a page of pages tab will insert the tab onto the end of that page's list
  • dragging a page up a notebook level will work ok
  • dragging a notebook into itself will do nothing and not crash the client :^)
  • new pages that come from the main gui level (such as from pages menu or file import) now open in the deepest open notebook (previously, they would always appear in the top row)
  • fixed some misc page of pages bugs
  • fixed a bandwidth calculation that meant 1s time delta rules were working at 50% capacity (e.g. 1rq per 1s rule for domains were actually running at 1rq per 2s)
  • improved a bandwidth estimate calculation that was cutting out early in some situations for large time deltas
  • tag manager's page up/down shortcuts no longer mistakenly navigate the archive/delete filter
  • added a network timeout option to the 'connection' options page
  • reduced some hover window show/hide flicker when the media viewer is fullscreen and the OS has a taskbar that pops in for non-fullscreen windows (this mostly affected Linux)
  • fixed a long-time issue with yes/no dialog layout
  • improved some rendering of EXIF-rotated and -flipped jpegs. I expect to add more support for this (and retroactive image metadata-parsing to figure out correct reversed resolution--atm rotated images remain stretched) in the coming weeks
  • fixed an error when cancelling the booru-picker dialog from the page chooser dialog
  • if no files or all files are selected, the 'invert' select choice will no longer be shown (in this case, it redundantly does the same job as 'all' and 'none')
  • fixed an issue where setting namespace sort as default would persist through a client reboot
  • fixed an issue where force idle debug mode was not waking sleeping daemons
  • increased frequency of mappings processing reporting
  • wrote an exception and the needed maintenance code for the 'repairfilesystem' dialog to allow a proceed action if the only remaining incorrect paths are thumbnail paths--in this case, the client will create empty prefix folders and prompt the user to regen thumbnails
  • export filenames are now clipped to make <255 total characters in path
  • removed the old 'processing phase' option, which was no longer in use
  • removed the proxy settings from the 'connection' options page--the new engine does not use this old system, but if there is demand, a more flexible system will return
  • misc image rendering pipeline updates
  • misc improvements

• version 269

  • nested pages now supported!
  • moved all page management (session load/save, new/close page, page navigation, page name maintenance, etc...) code from the main gui to the new PagesNotebook object
  • expanded the session object to hold nested page information
  • added a 'page of pages' page to the 'special' new page entry!
  • numerous other gui-notebook page-related event fixes and improvements
  • figured out cross-platform menu and other mouse event support for nested notebooks, but there may still be holes--please let me know if your new pages ever appear in the wrong tier!
  • ways to move pages up and down rows will come in the coming weeks!
  • main gui status bar now shows total bandwidth this session as well as current speed
  • added a cog button to network job controls that allow for manual- and auto-bandwidth-override
  • added a 'blocked?' column to the review bandwidth panel to quickly see which network contexts are currently available to do work
  • added a button to reset 'default' and 'global' bandwidth rules to review bandwidth panel
  • merged the network request start test and consumption code into one transaction, stopping some accidental overconsumption when the engine was under heavy load
  • did some logic work to make sure unusual network context rule/usage situations are visible on the review bandwidth panel for editing
  • network jobs now report more information as they get ready to start, including while they are held up waiting for a download slot
  • added a simple 'network profile mode' to the debug menu that atm just prints a summary of new jobs
  • fixed an error that could sometimes be a crash when 'review services' was opened while no pages were open in the main gui
  • fixed the pixiv login test button with a hacky workaround--I will make extensive proper login testing gui when I move to the login engine
  • fixed an issue in the youtube downloader
  • if the bandwidth or session managers are missing on boot, empty defaults will be created in their stead
  • bumped the max page limit up to 150--I expect to increase it more in the coming months as I rejigger how some gui stuff is laid out
  • renamed 'sort by age' to 'time imported'
  • fixed and improved some test code
  • cleaned up some shutdown code
  • should have fixed a rare unicode conversion issue when printing to log
  • misc improvements

• version 268

  • split the sort dropdown into two, splitting the sort type and sort order
  • file sorting works more intelligently behind the scenes in several ways
  • added 'sort by number of tags' to sort options
  • session pages now remember their sort status!
  • sessions also more reliably remember their actual thumbnail order for all pages (typically, this matters for importers atm)
  • network job controls now report an estimate of how long they will have to wait for bandwidth
  • subscriptions will now show some 'no more bandwidth to download files' text if they have to stop because of bandwidth rules
  • subscription 'should I start/continue' testing now has a little padding to forestall some potential unexpected long delays in operation due to edge cases
  • the edit subscription dialog now uses the new file import status control
  • fixed an issue with hentai foundry filters not applying (they added some categories since the downloader last worked, breaking the POST form), hence hiding most non-vanilla results
  • finished the 'database->migrate database' dialog and its help and removed the 'under construction' labels. the new help page is also now linked from the standard index.html
  • fixed an issue where pages would sometimes never initialise (due to being queued up after an infinite job like the network engine's mainloop!)
  • improved how all long-job threads are spawned
  • improved some more thread scheduling logic that meant some long-term jobs could be stacked undesirably
  • expanded how the new listctrl updates and deletes its data
  • reduced flicker on the new listctrl update events
  • the file import status window now uses the new listctrl
  • the manage subscriptions dialog now uses the new listctrl
  • collections now track their tags more accurately
  • moved the 'delete original files after success' checkbox up to reduce misclicks on ok'ing the file import dialog and added a bit of red warning text whenever it is on
  • sped up some behind-the-scenes content processing for large pages
  • improved how the program cleans some things up during exit
  • the issue with some clients not clearing their exit splash screen until a mouseover event should be fixed
  • at 120 open pages, the client will inform the user about the approaching max page limit
  • by default, import folders no longer delete anything. default is to ignore original files in all cases
  • deleted some old unusued code
  • misc cleanup
  • misc improvements

• version 267

  • drag-and-dropping a 4chan or 8chan url onto the client will now automatically open a thread watcher for that url
  • fixed an issue where web domain or subscription network contexts that included unicode characters in their context data were unable to serialise and save to the db, causing error spam when the bandwidth manager attempted to save itself
  • subscriptions will now show a network job control in their popup as they do network work
  • subscriptions will cancel more reliably during gallery parse
  • subscriptions now have some 'delay' logic that will stop them sometimes restarting as soon as they are cancelled or otherwise have to stop mid-work
  • subscriptions will now tolerate up to five 'already in cache' urls per page parse until it considers the page 'already seen'. this is to catch the odd additional late insert and avoid the problem of a page updating and shuffling everything up one as the subscription walks through pages
  • maintenance modal popup messages should no longer appear if their jobs are very quick
  • queued up modal popup messages will no longer flicker their dialogs on a de-minimise
  • should have fixed an issue where modal popup messages could sometimes error out on a close attempt, locking the client's whole gui and requiring a force quit
  • fixed an issue where the modal popup message dialog was OKing on a close attempt of a non-cancellable job, despite presenting user text indicating otherwise
  • did a bunch of data work on pages
  • page tab names will now be clipped to 20 characters by default
  • pages now show (num_files) after their name by default, although you can set this for only import pages or turn it off completely
  • options for these new behaviours are in options->gui--and page names will update immediately on dialog ok
  • subscription and import folder 'show files' events (and a couple of other misc occasions) will now launch their page tab with the sub/folder name rather than a flat 'files'
  • wrote a new listctrl class with custom sort code that does not suffer from the 'crash when sorted while many pages open' bug. it also handles data in a simpler way for hydrus
  • review bandwidth listctrl now uses this new class and should now not crash your client. I will replace all the other listctrls with the new class over the coming weeks
  • temporarily, no old listctrls will auto-sort themselves (as this causes a crash for many users)! if you do not suffer from the crash, please sort them yourselves for now

  • reduced some large-scale gui import lag

  • reduced content processing CPU load on clients with many thumbnails open
  • massively reduced content processing CPU load on non-current pages
  • offloaded newly-imported file thumbnail detection and generation to a non-gui thread
  • migrate database dialog now lets you move the whole database and all portable locations, which requires a client shutdown
  • the raw url downloader will no longer have a problem with pasted url lists that include empty newlines
  • all the downloaders/importers now sleep on cleverer event objects so they will burn less idle cpu and wake as soon as they have new work to do
  • fixed and otherwise improved some gallery downloader timing logic
  • hdd import pages now use the file import status control
  • the thread watcher's controls now try to wrap in a single sizer. it uses less space, but might sperg out, let me know if it is a problem. I'll replace the whole watcher timing system in the new engine anyway
  • fixed a gelbooru (and possibly others) booru parsing bug that meant half of the pages in the gallery walk were being skipped (e.g. 'mogudan' was producing ~350 files when there were actually 690 in the list)
  • fixed some dialog panel layout scrollbar-cutoff in sevaral Linux places and perhaps elsewhere
  • cleaned debug menu a little and added a save 'last session' entry
  • fixed an issue where some kinds of media would error on notification of new url association
  • clients will now save small transaction progress reliably within ten seconds no matter how idle they are
  • deleted the old 'gui capitalisation' option--I never got around to expanding it beyond a handful of menu labels, it was always too much to work on for too little reward
  • improved support for certain broken videos--these will import ok, but full rendering might be borked to different degrees, so let me know how it goes.
  • cleared out some old content processing code
  • cleared out some old unused db data
  • removed all old 'waiting politely' download settings and gui code
  • misc prep work for wx update
  • misc gui code refactoring
  • misc downloader cleanup and timing tweaks
  • misc improvements

• version 266

  • converted gallery downloaders to the new network engine
  • greatly simplified how gallery downloaders report network activity and converted them to show the new network job control as well
  • subscriptions also work on the new engine but will not show network gui yet
  • hacked hentai foundry and pixiv login to use the new network engine
  • successful logins to hf or pixiv now print to the log
  • the new network engine now clears temporary session cookies after 90 mins of inactivity
  • gallery downloaders and subscriptions now use the new 'downloader instance' and subscription bandwidth rules. by default, this means downloaders will do small bursts every five minutes and that subscriptions will do at most 256MB per day
  • subscriptions' bandwidth use is now listed by name in the review bandwidth panel
  • subscriptions use a new bandwidth test to determine if they should start or continue based on current bandwidth limits. it should mean subs do a good bit of work and then stop when they are supposed to without ever waiting on bandwidth more than 30s or so
  • thread watcher and page of images now ignore bandwidth limits when doing their 'page' fetching part
  • gallery page fetching will ignore bandwidth rules (in order to stop gallery walk desyncs from having to wait a long time)--it will fetch one page per five seconds
  • the thread watcher and page of images importers now work on their files and page-checking simultaneously--also, the page of images will process its page queue at any time, not only when the current file queue is finished
  • the gallery downloader page now works on its files and gallery page parsing simultaneously!
  • wrote a 'file import status' control to better wrap up the import summary, progress gauge, and file import status button into one panel
  • thread watchers now use the new file status control
  • 'page of images' downloaders now use the new file status control
  • the gallery downloaders now use the new file status control
  • all network jobs will now retry up to four times on the BadStatusLine ConnectionError, which seems to be a TLS (https) negotiation timeout/remote termination
  • all requests on the new network engine will now timeout after ten seconds
  • they will also retry on generic timeout errors
  • popup messages can now be shown in 'modal' mode as a dialog that prohibits interaction with the rest of the client.
  • these will not boot while the client is minimised
  • database maintenance routines will now all publish messages like this
  • 'migrate database' panel now publishes a modal message when it does a file rebalance
  • rewrote the controller-side pubsub pipeline to respond faster and consume fewer program resources, particularly for the client
  • import pages now update themselves in a less spammy way behind the scenes, meaning more active pages can be open at once without them stepping on each other and clogging things up
  • simplified how importers set their status information in several ways
  • reduced a swath of pubsub spam related to content updates
  • improved how spammy small jobs are written to the profile log
  • reduced text flicker on all download pages
  • more misc pubsub improvements
  • reduced some gui update/refresh spam on hidden pages
  • cleaned up a bunch of database->gui message reporting and cleanup code
  • added close other/left/right pages to tab right-click menu
  • the top-right media hover window will no longer refit-flicker on a ratings change
  • wrote a new panel wrapper for listctrls that handles the underlying row of buttons in a neater way and automatically disables them if they are nullipotent (mostly, this means greying out 'delete' buttons when nothing is selected)
  • several listctrls use this new panel
  • when it is not strictly necessary, videos that are >30MB will no longer use the CPU-expensive manual frame count parsing
  • a problem where newly reloaded thread watchers could sometimes stick in a 1/1 initialisation state _should_ be fixed
  • fixed gelbooru url parsing (they stopped using the janky redirect.php urls)
  • fixed an issue that meant ipfs pin was erroring when trying to show gui-side
  • 'page of images' downloaders now protest if they are told to close while working
  • some small layout and status text fixes
  • adminside petition processing now has a 'flip selected' box to flip checked status of all selected contents
  • adminside petition processing contents chechklistbox now supports multiple selection
  • improved the reliability of some shutdown code
  • cleared out some old unused code
  • misc new control cleanup
  • misc fixes
  • more misc fixes

• version 265

  • the bandwidth engine now recognises individual thread watcher threads as a network context that can inherit default bandwidth rules
  • tweaked default bandwidth rules and reset existing rules to this new default
  • review all bandwidth frame now has a time delta button to choose how the network contexts are filtered
  • review all bandwidth frame now updates itself every 20 seconds or so
  • review all bandwidth frame now has a 'delete history' button
  • review all bandwidth frame now shows if services have specific rules
  • review all bandwidth frame now has an 'edit default rules' button that lets you select and set rules for default network contexts
  • review network context bandwidth frame now has a bar chart to show historical usage!
  • bar chart is optional based on matplotlib availability
  • review network context bandwidth frame now lists current bandwidth rules and current usage. it says whether these are default or specific rules
  • review network context bandwidth frame now has a button to edit/clear specific rules
  • rows of bandwidth rules and current usage, where presented in ui, are now ordered in ascending time delta
  • misc bandwidth code improvements
  • client file imports are now bundled into their own job object that generates cpu-expensive file metadata outside of the main file and database locks. file imports are now much less laggy and should generally block the feel of the ui much less
  • removed the database 'rebalance files' menu entry
  • removed the 'client files location' page from options
  • db client_files rebalance will no longer occur in idle or shutdown time
  • (this stuff is now handled in the migrate database dialog)
  • 'migrate database' now uses a dialog, meaning you cannot interact with the rest of the program while it is open
  • migrate database now has file location editing verbs--add, remove, +/- weight, rebalance_now. thumbnail location and portable db migration will be added next week
  • flushed out the backup guide in the getting started help, including to reflect the new internal process
  • the client now saves the 'last session' gui session before running a database backup
  • the shutdown maintenance yes/no dialog will now auto-no after 15 seconds
  • gave status bar tabs a bit more space for their text (some window managers were cutting them off)
  • tumblr api lookups are now https
  • tumblr files uploaded pre-2013 will no longer receive the 68. subdomain stripping, as they are not supported at the media.tumblr.com domain (much like 'raw' urls)
  • pages will now not 'start' their download queues or thread checkers or whatever data checking loops they have until their initial media results are loaded
  • key events started from an autocomplete entry but consumed by a higher window (typically F5 or F9/ctrl+t for refresh or new page at the main gui level) will no longer be duplicated
  • fixed a shutdown issue with network job controls that could break a clean shutdown in some circumstances
  • if the user attempts to create more than 128 pages, the client will now instead complain with a popup message. Due to OS-based gui handle limits, more than this many pages increasingly risks a crash
  • if the client has more than 128 pages both open and waiting in the undo menu, it will destroy the 'closed' ones

• version 264

  • converted the page of images downloader to the new network engine
  • converted the thread watcher downloader to the new network engine
  • rejiggered page of images and thread watcher management panels to better separate the two loops they work on
  • rejiggered some more downloader layout stuff in prep for a new control to better display url cache and overall import status--downloader-global pause button is now always at the top, for instance
  • removed 'waiting politely' indicator from download pages that use the new network engine
  • network job controls now have a simple border
  • network job controls will show current bytes progress and speed in a more intelligent and useful way--no longer spamming the initial 0KB/s, for instance, and flashing more useful summary information when downloading many small files
  • network job controls now display their current bytes progress and speed on a separate right-aligned text beside the main status text (to stop the speed text jumping around so much)
  • network job controls will not show the current download speed until some bytes are actually read
  • wrote a first version of 'services->review bandwidth' review frame, which lists all network contexts with data in the past month
  • wrote an independant and live-updating review frame for indivdual network contexts, launched from the review bandwidth frame, which shows current and historical bandwidth use for that context
  • the main gui status bar now reports current bandwidth usage! updated once a second
  • database->backup process now has some nicer gui, better and more reliable workflow, improved file copy notification, and now correctly obeys a popup-cancel command
  • started the new 'migrate database' gui under the database menu. it currently shows live usage and will gain some verb buttons to atomically alter things hopefully next week
  • moved from a strict begin/commit db model to a save/release system that only commits changes to disk at most every ten seconds or so. small 'write' transactions will now often have no disk-sync overhead and so are super fast, particularly when they come in a large batch.
  • simplified database connection, disconnection, begin, commit, and rollback code significantly
  • the url table in the client database is now non-unique (i.e. multiple files can have the same url). this may lead to undesired 'already in db' or 'deleted' statuses in some downloaders as they check for url status before starting their downloads. if this change turns out to a be a big pain in the future for pixiv manga or whatever, it may be revisited.
  • fixed a bug in deviant art parser that sometimes meant nsfw urls could not be found
  • fixed an issue where progress gauges could sometimes get stuck in a pulsing state
  • fixed a bug with the new optimised result building which sometimes occured when all the files in a batch had common ipfs service membership
  • improved support for unusual videos
  • improved accuracy of duration calculation for unusual videos
  • improved database optimisation after initial and ongoing repository processing
  • a 'wake subscriptions' event now occurs even on 'manage subscriptions' cancel (meaning temporarily-dialog-paused subs will always continue after the dialog is closed)
  • added a careful manual commit to stop the possibility of the definition-desync some users noticed after a power-loss during a big repository sync commit
  • removed some redundant old db compatibility code
  • permitted sqlite installations that allow in-memory temporary tables to use them (but not for vacuums, which tend to be a bit big for this stuff)
  • cleaned up some redundant timer id code
  • misc improvements
  • misc refactoring

• version 263

  • greatly improved how gui sessions are loaded--now the page tabs are loaded instantly, but the thumbnails are loaded in the background. session loaded should be significantly less laggy and buggy
  • the issue of pages sometimes initially sizing at zero size (which could be caused by minimising the client while a session load was ongoing) should hence be better/fixed completely!
  • gui sessions will now load their files in the exact order in which they were saved--behaviour that I think was not previously always reliable
  • more general network code work and polishing
  • added and improved unit tests for network code
  • improved how short-time-delta data bandwidth is reported
  • improved how short-time-delta data bandwidth is tested
  • wrote a networkjobcontrol to display and control the new network job object
  • tumblr parser now produces 68.-less urls
  • tumblr parser now produces https urls
  • cleaned up tumblr parser a little
  • url caches will clip existing tumblr urls of the 68.-subdomain and convert to the new raw format, clearing out dupes along the way
  • url caches will convert existing tumblr and pixiv urls to https, clearing out dupes along the way
  • the pixiv parser now deals with missing creator/title tags without errors
  • extended the specific file domain tag cache to also store deleted mappings, resulting in much faster request building for clients with large numbers of deleted mappings
  • improved some downloader page queue text display timing
  • added support for more types of mp4 file
  • improved how some memory maintenance calls work
  • improved how hydrus datacaches track their recently-used-data fifo list
  • pages now regularly clear out spare thumbnail canvas bmps
  • pages now regularly clear out cached thumbnail canvas bmps when they are not the currently viewed page
  • import caches, when asked for url membership, will now test both the http and https versions the url
  • maybe improved how 'open in file browser' works in windows
  • fixed the 'recount video frames' advanced thumbnail menu entry, which wasn't working with the new ffmpeg wrapper
  • moved some bloaty master hash data out of client.db and into client.master.db
  • pubsub profile will no longer profile 'message', as it just makes for spam
  • reduced some serverside pubsub spam
  • reduced some significant clientside pubsub spam that I think was smashing the gui event loop at inconvenient moments
  • improved some client shutdown object sync code
  • fixed an issue where some duplicate maintenance popups would not clear themselves up properly if interrupted mid-job
  • cleaned up some http-https conversion and comparison code
  • fixed some status-setting code that meant thumbnail pages were sometimes setting status after they were replaced and scheduled for deletion
  • misc improvements

• version 262

  • added apng support!
  • sessions will now append from the default insertion index (e.g. right of current page)
  • sessions will now load in the correct order if the default insertion index is to the left of the current page!
  • fixed an issue with custom 'favorites' gelbooru parsers
  • manage parents/siblings dialogs now load their larger datasets on a thread, after initialising
  • sped up how some default thumbnails are generated on client boot
  • sped up how resized thumbnails are generated
  • refactored ffmpeg video parsing to be simpler and easier to maintain
  • improved some ffmpeg parsing to be much faster
  • ffmpeg can now parse video that reports no duration
  • fixed some more bad video framerate parsing
  • the 4chan/8chan thread watcher will now always generate https urls (even if you enter an http thread url)
  • pixiv will now produce https urls
  • the known url file status system will now check both http and https versions of an url it is given
  • refactored some core elements to simplify common controller requests
  • started on some database migration help and ui--should be finished next week
  • flushed out network engine a little
  • more refactoring and simplification of new network engine object coupling
  • flushed out bandwidth manager
  • misc logic tweaks in bandwidth management
  • wrote a network session manager
  • wrote a new class for identifying all future 'network contexts'
  • employed NetworkContext everywhere in the new engine, massively simplifying several things
  • wrote the first batch of tests for my new network engine and its network jobs
  • wrote tests for the bandwidth rules object, including covering the new short time delta support
  • improved reliability and speed of bandwidth unit tests
  • wrote tests for the new domain-based bandwidth manager
  • misc improvements
  • more misc improvements

• version 261

  • wrote a new manage dialog for urls
  • added 'manage known urls' to media right-click menus!
  • double-left-clicking on any video animation will 'open externally'!
  • added an option to options->gui to set where new pages will appear by default--either far left/right or left/right of current page
  • the tumblr parser now produces '_raw' urls when the post was posted in 2013 or later
  • created a new 'number' subtag cache that will be populated on update
  • created a new 'tag as number' system predicate that can search for, say, all 'page:' tags > 200
  • bandwidth management now tracks requests and num_bytes more sensibly
  • bandwidth tracking objects can now better handle bandwidth usage and rule application in short intervals (i.e. sub-5-second) (however, the current networking engine cannot yet use this information accurately)
  • wrote unit test for bandwidth tracker, including for the new short interval timing
  • fleshed out the new network engine
  • fleshed out the new network job
  • thread watchers can now have a time delta that includes days (giving a max check period of 360 days, wew)
  • adminside mapping petition processing now has adaptive max total petition weight--lower file count range petitions will have much higher max total permitted weight
  • added a new 'callto' debug reporting mode that reports on current thread pool jobs
  • improved calltothread pre-spawning checks to reduce outside chance of deadlock in busy periods
  • the advanced review services repository panel buttons are now hidden unless in advanced mode
  • eliminated some animation buffer looping redundancy
  • fixed a little animation next-frame prediction code
  • fixed up some '1 minutes'-type time_delta->string conversion
  • fixed up label on time delta control button
  • fixed some shutdown thread interactions
  • fixed an issue where sometimes empty tags could be entered into the manage tags panel
  • added some pydeadobject error handling during client shutdown
  • refactored some db multi-tag->file search code, cleaned up wildcard searching
  • misc string cleanup
  • misc dialog cleanup

• version 260

  • fixed video parsing when the video metadata includes random non-utf-friendly garbage
  • fixed video parsing when ffmpeg reports no fps at all
  • improved video frame counting accuracy
  • thumbnail waterfall process now adapts to the current speed of the cache and can hence use significantly less overhead
  • the thumbnail fading process now adapts to the current rendering speed, delivering smooth fade when CPU is available but otherwise skipping frames and more reliably filling in thumbnails within a quarter of a second
  • canvases now draw their thumbnails with slightly less overhead
  • increased database synchronous pragma to FULL to better safeguard against power loss during multiple-db commit--we'll see if it slows things down too much, and maybe add an option about it
  • cleaned up and improved some client gui pubsubs
  • made pubsub profile mode far less popup-spammy
  • if a profile takes less than 20ms, it now won't be fully printed to the log
  • tweaked some more server object maintenance code
  • added 'changelog' link to help menu
  • updated lz4 library and fixed some old deprecated calls
  • misc serverside pubsub cleanup
  • misc fixes
  • misc refactoring
  • misc code cleaning

• version 259

  • planned out new networking engine and started the principal objects
  • renamed the 'exact match' duplicate status to 'same quality', to reduce confusion on what is appropriate for this status
  • the duplicate filter, on hitting the delete key, now offers the option of deleting both files
  • the duplicate system now combines duplicate status setting and the consequent batch of content updates into the same database transaction, speeding things up
  • the duplicate system now batches multiple duplicate status setting into a single transaction, massively speeding up large filters or thumbnail status set actions
  • misc duplicate help tweaks
  • you can now edit the default duplicate merge options from the new thumbnail duplicate menu
  • the duplicates page's jobs are less demanding on gui time and take better breaks if something else happens
  • renamed the new dupe system predicate to 'system:num duplicate relationships' to clarify what it searches
  • for normal queries, current and pending mappings are now fetched from a faster mappings cache. you should see faster result building across the board, particularly on fresh boots or otherwise slow-disk-access systems
  • added a prototype 'advanced mode' (defaulting to off, so experienced users will want to turn it on) under the help menu that will enable menu items that are often not helpful to new users. I will add more things to this in future, suggestions welcome
  • the new thumbnail menu dupe relationship set stuff is now considered advanced
  • thumbnail menu find similar files is now considered advanced
  • the thumbnail menu copy hash entries are now considered advanced
  • advanced content update buttons (on manage tags and review services) are now considered advanced
  • added an advanced mode 'open file location' entry to the thumbnail share menu, which will open the file in your OS's file explorer (not available on Linux)
  • added an advanced mode 'correct video frame count' thumbnail menu entry that will force-apply last week's more accurate video frame counter to correct and videos that render too fast and then cut off
  • fixed many entries on the media viewer menus, which were being blocked by an over-eager 'can continue' test and hence silently failing
  • fixed the youtube downloader on Linux and OS X--both now use youtube-dl
  • fixed an issue where the GetLighterDarkerColour function was producing very bright alternates to very dark colours (meaning dark dupe filters were having their bright background text rendered unreadable)
  • improved video frame number parsing accuracy
  • improved the accurate version of video frame number parsing accuracy, particularly for longer videos
  • the network engine now reports 5xx http status codes as ServerException to better contextualise to the user what went wrong
  • adminside mapping petitions are now sub-ordered by tag
  • adminside sibling/parent petitions are now ordered by the 'older' tag and sub-ordered by the 'newer' tag
  • censorship taglists are now roughly sorted
  • reduced default shutdown work max time to five minutes
  • improved how subprocesses are started
  • misc cleanup
  • misc refactoring

• version 258

  • added a duplicate entry to the thumbnail right-click menu
  • the new duplicate menu will now attempt to fetch known duplicate counts for the focused file for the current file domain (if the db is locked, it will say so rather than block)
  • the new duplicate menu's counts in the thumbnail menu can be clicked to show those files in a new page
  • the new duplicate menu allows you to set the four main dupe statuses (with default merge options), or a customised verson of the same, or set unknown/potential, or delete dupe status completely, for all the possible pairs in the current selection
  • added tentative support for variable framerate files, manually counting up their frames and displaying them with an averaged constant framerate in the client renderer. this now permits a number of webms that were previously 'mime unsupported'. let me know how it works for you!
  • cleaned some server locking code, hopefully fixing the ssl handshake issue (which is actually a deadlocking issue)
  • export file dialogs will remember the last value of 'export tags to .txt files?', and clicking that checkbox will always launch the dialog for editing (rather than clearing if there are some set)
  • fixed the advanced content update (as used by service-wide update), which was not able to differentiate namespaces since the recent service changes. I apologise to anyone affected by this--I will add some unit tests to make sure it doesn't happen again
  • reworked and cleaned some canvas event handling code
  • the canvas will now accept mouse wheel events even when it does not have focus
  • the canvas frame will no longer accept keyboard events (and hence activate fullscreen_switch) if the mouse is over a flash window
  • hydrus servers now respond to /robots.txt with a 'disallow all'
  • servers will now provide mappings petitions to admins in groups of similar size, rather than mixing petitions of weight 5000 with a hundred of weight 1
  • fixed the back/skip buttons in the archive/delete filter's top hover window
  • the file path tagging dialog's namespace and regular regexes now support parenthesis groups, and in that case will take each of those submatches instead of the wider match
  • hence regexes with groups will now compile in that dialog
  • you can now 'new page here' on the final (rightmost) page tab
  • fixed an issue where media-viewer-launched manage tags frames would yield focus to the main gui if they had opened a sub-dialog while they were open
  • the tag censorship edit panel now has a 'help here -->' label in blue text
  • file imports will fail properly when the final temp_path->client_files file copy operation fails (due, for instance, to the destination being on a removable medium that was recently disconnected)
  • fixed serverside upnp maintenance, which was silently failing early, and also updated it to the new server object code
  • wrote a new checklistbox dialog that handles the underlying data explicitly rather than messing around with external text->data maps
  • replaced all the old string checklistbox dialog instances with the new one
  • subscriptions will now correctly clear their 'recent error' status on a successful sync
  • refactored and cleaned how basic mediaresult objects store and consult simple file information
  • locationsmanager objects now handle inbox status and the related content update events that affect it
  • fixed a bug in HydrusTagArchive when attempting to figure out hash type from an existing hash
  • removed the obsolete servertoclientcontentupdatepackage object
  • misc duplicate code cleanup
  • misc serverside service code improvements
  • misc small fixes
  • misc cleanup

• version 257

  • the duplicate filter will now maintain zoom on files with the same ratio
  • split the duplicate merge options into separate tag/rating controls--you may see some duplicate service entries, but these will be cleaned on your next shutdown
  • duplicate merge options now allow syncing 'archive' status
  • duplicate merge options now allow 'delete both files', which you may find use for in custom actions
  • created a tag censorship object to handle and action a rich tag censorship ruleset
  • tag duplicate merge options now have and use this tag censorship object to filter which tags are merged, with an initial value of 'let everything through'
  • wrote a tag censorship edit panel and tied it into the duplicate action edit panel so these new tag censorship objects can be edited
  • added an optimisation to the duplicate status setting code--if two files are better/worse, they are inherantly duplicates and so 'not dupe' and 'alternate' relationships apply to both equally and can be duplicated
  • fixed and culled and normalised the 'this has more tags' dupe filter statements to be more accurate and useful
  • added 'this has a larger filesize' type-statements to the dupe filter
  • created a 'system:duplicate relationships' predicate that can find files based on how many duplicate relationships of a particular type they have
  • cleaned up some misc duplicate filter code and added some tooltips to the top hover window dupe action buttons
  • added 'move to left/right end' to main gui page tab right-click menu
  • added 'new page' and 'new page here' to main gui page tab right-click menu
  • you can now right-click for a menu from empty tab space on the main gui
  • the main gui statusbar now updates more efficiently when under heavy refresh load
  • the main gui statusbar now shows db read/write/commit status and sets the current db job summary as its tooltip--if you experience persistent hangs, please hover over the statusbar and report what you see!
  • export tags to .txts checkbox will now default to 'all services on' when checked
  • fixed the thread watcher, which was accidentally disabling its text input early
  • 'similar files' searches launched from the thumbnail menu will now default to 'my files' file domain rather than 'all local files'
  • downloader pages will now correctly sort their files on initialisation
  • refactored and generally cleaned up some collect and sort code
  • fixed some unlikely-but-possible collect/sort bugs
  • fixed some bad layout in the top-right hover window that was making it grow unreasonably tall when many urls were shown
  • on the different download import pages, the progress gauge that shows file download progress will now reset back to 0 as soon as the file download is complete
  • fixed a problem where video imports with unicode characters in their path were failing to mime-parse
  • improved the file import status update pipeline to better deal with large transactions (like skipping/deleting/retrying a thousand rows at once). all these big transactions should lag the gui far less
  • improved some misc import status cache code
  • made first step in a big size rewrite job that will size many elements according to local system font size rather than specific pixel values
  • hydrus servers now explicitly default to TLSv1.2--we'll see if that clears up some of the handshake timeout problems we have recently seen
  • cleaned up a bunch of possible pydeadobjecterrors when the new review services panel is closed
  • improved and rescheduled gelbooru redirect url purge
  • added a catch-and-recovery to hydrus network session initialisation, which may sometimes receive invalid data after a service deletion
  • added similar catches to tag parent/sibling initialisation, which apparently can be vulnerable to a similar invalid data problem
  • I think I cleaned up some more Linux ClientToScreen console errors
  • refactored and cleaned some frame size event responsibility
  • refactored and cleaned the panel and controls that display file import status
  • did a little more menu code cleanup
  • misc cleanup
  • misc fixes

• version 256

  • the duplicate filter now loads new pairs off the gui thread. it will display 'loading pairs...' during this time
  • media viewers of all kinds are now more comfortable displaying no media (when this occurs, it is usually a frame or two during startup/shutdown)
  • the duplicate filter now responds to any media_viewer_browser navigation commands (like view_next) with a media switch action
  • you can now alter the duplicate filter's background lighten/darken switch intensity from its top hover window's cog icon
  • fixed a bug in the new dupe pair selection algorithm that was preventing pairs from being presented as groups
  • the duplicate filter will now speed up workflow by automatically skipping pairs when you have previously chosen to delete one of the files in the current batch
  • auto-skipped pairs _should_ be auto-reverse-skipped on a 'go back' action
  • added a |< 'go back' index navigation button to the duplicate filter top hover window
  • the duplicate filter now displays several 'this file has larger resolution'-type statements about the currently viewed file. it lists them on the top hover window and in the background details text underneath
  • the duplicate filter _roughly_ attempts to put the better file of the two first. this will always be indexed 'A'
  • the duplicate filter now shows done/total batch progress in its index string--not sure how clear/helpful this ultimately is, so may need to revisit
  • an unusual bug where Linux would spam the 'All pairs have been filtered!' duplicate filter message over and over and then crash _should_ be fixed--the filter no longer waits for that message to be OKed before closing itself
  • drag-and-dropping text onto the client will now a) open a url import page if none already exist and b) put the dropped text into the input box of the first open url import page (and focus it, so you can quickly hit enter)! This works when dragging text links from browsers, as well
  • you can now 'append' gui sessions, which will just append that session's tabs to whatever is already open--say, if you have several 'favourites' pages you want to be able to quickly load up without having to break your existing workflow
  • ipfs services now have a 'check daemon' button on their review services panel which will test the daemon is running and accessible and report its version
  • fixed the 'test address' button for ipfs services on their manage services panel
  • the client can now automatically download files it wants and knows are on an ipfs service
  • middle-click on an 'all known files' domain thumbnail will now correctly start a download (as long as a specific remote file service is known)
  • the multihash prefix option is reinstated on ipfs manage services panels
  • the gelbooru parser now discovers the correct page url to associate with its files
  • wrote some redirect fetching code to fix the gelbooru bad urls issue
  • discovered a quicker fix for the gelbooru issue--the redirect location is the garbage in the original url in base64
  • all downloader/subscription url caches will purge any old gelbooru 'redirect.php' urls on update
  • fixed an issue where 'previously deleted' gallery/thread imports were returning 'fail'
  • fixed a problem that was causing some redundant laggy work in adminside petition processing
  • thread watchers will now remember their file and tag import options through a session save even when no thread url has yet been entered
  • fixed an issue where media 'removed' from a media viewer view of a collection resulted in the entire collection being removed at the thumbnail level
  • fixed an issue where media deleted from a media viewer view of a collection resulted in the media not being correctly removed from selection tags
  • tag, namespace, and wildcard searches on a specific file domain (i.e. other than 'all known files') now take advantage of an optimisation in the autocomplete cache and so run significantly faster
  • fixed a hover window coordinate calculation issue after minimising the media viewer on some platforms
  • removed some 'all files failed to download' spam that could sometimes occur
  • misc fixes

• version 255

  • the duplicate filter now supports shift+left-click to drag, like the archive/delete filter (this remains hardcoded for now)
  • if a pair in the dupe filter has the same resolution, they will now maintain zoom and pan when switching back and forth (I might increase this to work for same ratio as well, let me know how it works in the real world)
  • the duplicate filter will show a lighter/darker background colour as you scroll the current pair
  • the way lighter/darker and alternate lighter/darker colours are calculated is now centralised and should be more reliable in edge cases
  • improved the dupe filter pair selection algorithm--it now chooses pairs more reliably under edge case conditions and prioritises decision-groups that have high potential decision value. it should also run a little faster
  • increased the dupe filter batch size to 250, let's see if it causes any problems
  • the close button on the dupe filter's top hover window now works
  • fixed the duplicate filter disappearing pairs that were skipped (meaning you could not go back to revisit them)
  • fixed a frequent deadobject error when the dupe filter closes
  • the shutdown 'maintenance due' test is less sensitive to dupe search tree rebalancing, which typically only takes half a second
  • the archive/delete filter now uses the new shortcuts system for both keyboard and mouse input
  • the archive/delete filter now intercepts archive or delete commands from different sources ('media' shortcuts, top hover frame button presses) more reliably and converts them into filter actions (hence moving on to the next file)
  • you can now move pages one to the left or right from their menu!
  • on the top-right hover window (and the background underneath), ratings are now on the top, and hence will always be in the same location as you scroll through your media regardless of known urls, remote location, or inbox status
  • autocomplete searches for tags with apostrophes, quote marks, braces, brackets and paretheses should be more reliable
  • urls are now associated with files through the same 'content' pipeline as tags and ratings and so on
  • gui-level media is now aware of the 'new url content update' event and will update and redraw itself appropriately
  • fixed deviant art nsfw parsing, but it might not hold for long. proper fix here is to wait for the downloader overhaul
  • networking engine now uses the 'requests' module's CA .pem (which the overhaul will be moving to anyway), which should reduce the frequency of ssl verify failures (gelbooru on the relatively new 'Let's Encrypt' CA had this problem for many users)
  • fixed the networking engine redirect parsing for gelbooru's unusual location header. unfortunately, gelbooru is still giving mickey-mouse garbage redirect urls from its main thumbnail pages, for which in this engine there is no immediate fix
  • neighbouring .txt tags will now be properly cleaned and sibling-collapsed in the path tagging dialog
  • neighbouring .txt tags will now be properly sibling-collapsed in the import folder workflow
  • making a media viewer borderless fullscreen and back will now recenter the media (previously, the current drag delta was not reset, so this frequently put media off-screen)
  • fixed a serious issue where the media viewer could lock the client up on opening with a video if its gui options set 'remember size' to false
  • tag import options objects will cleanse themselves of missing services on options save/client shutdown
  • manage tags now defaults to cross-referencing 'my files' on the 'local tags' domain, where 'remote' (i.e. deleted) files' tags are not useful
  • reduced memory use when importing large pngs with transparency
  • improved adminside petition processing gui reporting
  • servers will now cap the size of their mapping petitions so as not to ovewhelm the admin processing them (they now won't be both >20 tags and >1000 total row weight)
  • the media viewer's manage tags frame now listens for content updates from outside, so if you alter an in-view file's tags (such as with a shortcut key), the manage tags dialog will update as it happens
  • moved a number of the buttons on the top hover windows to the new unified internal command engine (which the new shortcut system also uses)
  • disk cache maintenance uses fewer resources but now occurs in the foreground (and should hence more reliably maintain the cache)
  • critical repository service id lookup errors will now automatically reset the repository's processing cache and better inform the user of what has happened. if you see this, please let me know the details and how this error fired in the real world
  • updated and reinstated the ipfs service panel in review services
  • updated some ipfs service code for the new service system
  • misc refactoring and cleanup
  • more cleanup and deletion of redundant old pubsub command code
  • updated ubuntu build machine to 17.04 and opencv 3.2

• version 254

  • the duplicate processing page is now on the new page picker, under the new menu 'special'
  • improved the dupe pair selection algorithm--it should be faster for everyone
  • added help icon to dupe processing page--it can launch of a couple of message boxes with simple help, or you can open some html help
  • wrote some dupe processing html help!
  • the dupe processing page now refreshes its numbers when the dupe filter closes
  • the archive, inbox, delete, and undelete buttons on the duplicate filter's top hover frame now work
  • delete and shift+delete keys now work to delete/undelete a file in the duplicate filter
  • the mouse cursor now hides/shows on halt and new movement on the duplicate filter
  • you can now set 'content' shortcut actions (adding tags or ratings) to shortcuts in the 'media' reserved shortcut set (which are always on in media contexts), although they don't yet work in the thumbnail view yet
  • added a help button to the manage shortcuts panel
  • the edit shortcuts dialog will now now allow you to name a custom shortcut set to one of the reserved names and will explain the problem in a message box
  • import pages update more efficiently during periods of busy cpu
  • import pages will use less idle cpu time generally
  • import pages will adaptively use less cpu time when they are in the undo deletion queue or have no files to import
  • the page of images downloader will spam a little less idle time
  • the canvas background details and the top-right hover window will now only show 10 urls max
  • added a help button to the tag import options panel to better explain the namespace selection and explicit tags
  • added 'gui report mode' to the help menu, which will report key and mouse shortcut events and and matched commands
  • the adminside petition panel now sorts multiple petitions by number of files descending
  • misc improvements
  • misc help cleanup
  • misc fixes

• version 253

  • created a new object to hold tag and rating merging and 'worse file' deletion options
  • wrote a dialog to edit this new object
  • established some simple defaults for this object for different duplicate status actions
  • the cog icon on the duplicates filter now lets you edit these defaults
  • the 'custom' duplicate filter action now works--first by asking you what status you want to set, and then by throwing up the new merge options dialog to tune it to whatever you like
  • wrote comprehensive unit tests for the new object
  • fixed the super slow dupe filter launch time problem
  • added a 'known urls' submenu to thumbnail and browser canvas right-click menu that lists all known urls for a file with an option to launch or copy the url
  • added known urls' hosts to the top-right canvas details background, just below where known file repos are listed
  • added the same known urls' hosts as clickable hyperlinks to the ratings hover window that pops over that top-right area
  • added 'delete from deleted files' action to the local tags's service-wide update panel. it will limit the deletion to mappings that are currently on files that have been previously completely physically deleted from the program
  • fixed namespace filtering on service-wide update panel
  • the hentai foundry downloader broke, so the update code will pause all hf subs. the solution is not trivial (it is part of the downloader overhaul), but I will try to fix it soon
  • debuted a new question-mark help button to better explain .txt tag importing on the manual import tagging dialog and the manage import folders dialog
  • fixed a small potential error due to bad parsing in the 'page of images' downloader
  • fixed a typo bug that stopped the 'delete shortcuts set' action working in the manage shortcuts dialog
  • I may have fixed an issue where the server was sometimes not shutting cleanly with a keyboardinterrupt
  • fixed the media embed button not reliably updating its thumbnail
  • fixed an issue where a dummy animation bar was displaying on embed buttons that showed static images that included transparency
  • the serious db missing tag and hash states will now not throw an error but will inform/spam the user (and hence not prohibit a boot)
  • attempting to open a second manage tags frame from the media viewer will now instead put the focus on the first (previously, multiple manage tags frames could be made)
  • misc db code cleanup that should result in faster result building in certain situations
  • misc improvements
  • misc layout fixes

• version 252

  • the duplicate filter now processes pairs in batches and hence supports 'back' actions to revisit decisions. you will be prompted every fifty or so pairs to commit and checkpoint your progress
  • the duplicate filter now presents related pairs together, rather than picking at random
  • fixed a bug in duplicate filter shortcuts initialisation
  • simplified duplicate filter default shortcuts (these will be overwritten on update) to only use simple left- and right-click for 'this is better' and 'alternates', since those are by far the most common actions. middle-click now goes back, like the archive/delete filter
  • converted old 'main shortcuts' system to the new shortcuts system, also splitting it up into 'media', 'main_gui', 'media_viewer', and 'media_viewer_browser' constituent parts that will be applied in different contexts
  • because the change is so significant, all clients will have their old 'options' shortcuts reset to the new default--I expect to expand shortcuts further in the next few weeks, so this default-overwriting will likely happen again, so you will likely wish to wait before recustomising your basic shortcuts
  • completely eliminated the old main shortcuts system--all references now bodge with the new system to varying neatness
  • as the old main shortcuts system no longer exists, the 'shortcuts' page on file->options is gone--all shortcuts are now managed through file->shortcuts, which is a completely revamped version of the custom filter shortcuts editing dialog
  • selecting shortcut commands is significantly simpler for the reserved shortcut sets
  • all media viewers with hover windows now have a 'keyboard' shortcuts icon button on their top hover window--it links to the manage shortcuts dialog as well as the current active custom shortcut sets and default custom shortcut sets
  • as the above system supercedes the old custom filter system, custom filters are completely removed from the program! your existing custom shortcut sets will survive, but you probably want to purge them of all the redundant junk they still have
  • all the media canvases use the new shortcuts system
  • the canvas frame uses the new shortcuts system
  • the thumbnails canvas uses the new shortcuts system
  • the main gui uses the new shortcuts system
  • some other misc places use the new shortcuts system
  • most simple shortcut command actions have been renamed to be more readable
  • some shortcut actions, like zoom stuff, is no longer hardcoded!
  • mouse shortcuts are still not widely supported!
  • the shortcut command edit dialog now throws a veto-driven error message if you try to ok on an invalid command (a blank services choice or action string, that sort of thing)
  • ctrl+r is now 'remove_files_from_view' by default. the old 'show_hide_splitters' is now ctrl+shift+r by default
  • the preview canvas, if focused, now responds to many normal media viewer shortcuts (content stuff like archive/manage tags, and canvas-specific like frame back/forth, zooming and panning)
  • the new shortcut system now interprets double-clicks of any mouse button to be a second single click
  • the new shortcut system correctly 'flips' ratings on and off, rather than always 'setting' to the chosen value. optional 'set only' support will come in the near future
  • more shortcuts will correctly and reliably propagate to canvases when any part of a hover frame has focus
  • fixed a conflict between the new shortcut system and taglists, which were no longer accepting otherwise interesting keys, like enter
  • created a shortcuts manager cache that deals with a bunch of the shortcut workflow centrally
  • completely eliminated the old accelerator table/menu command system for the main gui window
  • lots of misc shortcut-related work
  • fixed wildcard file search predicates that are on a specific file service and have no namespace (like 'mar*')
  • certain routines that can cause mass refreshing of the menubar (like import folders) will now not spam (and often queue this spam up and hang the gui) the menu so much. the menubar will now always collapse multple overlapping refresh calls to reduce cpu load
  • fixed a focus-None issue in the new hover window focus detection code
  • improved some more focus detection and comparison logic--quick rating-scrolling in the media viewer should be less janked
  • improved 'touch' drag event detection and improved media canvas cursor hide/show logic in general
  • fixed '&' display in notebook page names
  • fixed '&' display in some common dialogs' text
  • wrote a new statictext class that deals with '&' better and in future will autowrap and maybe some other stuff and then switched most of the simple instances of wx.statictext over
  • updated some out-of-date server help r.e. admin service initialisation
  • converted a little of the help on custom filter->custom shortcuts. I'll do more in future
  • wrote a simple checkboxlist dialog
  • misc cleanup

• version 251

  • started shortcut overhaul by updating shortcut storage and underlying objects
  • shortcuts will now support multiple modifiers (ctrl, alt, shift) in some places
  • the regular shortcut entry control now supports mouse events through a radiobox (although only the duplicate filter will capture and deal with mouse events on the new system!)
  • on db creation or update, a new 'duplicate_filter' shortcut set will be generated that includes mouse shortcuts and new duplicate_filter commands
  • the duplicate filter now obeys this shortcut set
  • duplicate filter shortcut edit cog menu now works, and the active shortcut set will update if edited
  • wrote simple duplicate pair status update db code
  • added dupe filtering optimisation--given A > B, anything else better than A will be set to be better than B, and anything worse than B will be set to be worse than A
  • added dupe filtering optimisation--after any duplicate status change affecting A and B, any 'same file' siblings of A and B will receive the same relationship
  • the 'process now' button on review services is now gated by a yes/no dialog that better explains what's about to happen
  • the repository buttons on review services will disable when they can't do anything
  • repository update processing will now cancel mid-job much faster
  • reintroduced the 'service-wide update' button to tag services in review services
  • reintroduced the 'clear trash' button to the trash file service in review services
  • reintroduced num_files rating service reporting in review services
  • fixed mouse scroll wheel events from the ratings hover window not being correctly processed by the main media canvas
  • added some 'touch' event detection to try to better deal with media dragging through a touchscreen
  • import folders will now 'action' their import paths as they go, rather than only at the end of their import run
  • the new taglists are now better at remembering their selection through a content change
  • opening a file or path from a non-windows client should now create a non-child process for open/xdg-open calls that block (so closing the client should not then close the child process movie player application or whatever)
  • added unit tests for the new shortcut object
  • added unit tests for the new application command object
  • fixed unit tests for the updated shortcuts object
  • misc cleanup

• version 250

  • improved 'file storage locations' help page descriptive text, including adding statements for actual current storage percentages
  • improved boot missing file storage error handling
  • wrote a 'repair file storage' dialog panel to manually fix missing folders during the boot phase
  • added some bells and whistles to the repair file storage dialog. it now shouldn't be possible to boot the client with an invalid file store--let me know how this works for you
  • added 'apply all siblings to all services' to options->tags
  • whenever combined-service siblings are consulted, local siblings will now always have precedence (so, you can now definitely overwrite any ptr siblings you don't like at the local level). this stuff remains a mess, however. more work is needed
  • fixed searching for 'like' ratings
  • rating predicates will now render their value as 'like'/'dislike' or '3/5' rather than the underlying float
  • fixed a variety of predicate string rendering errors when 'show namespace' is false
  • fixed delete key not firing on autocomplete text entry controls
  • the autocomplete entry's text box will now take home/end/up/down events when there is only one result in the dropdown, when navigation of that list is not important (previously, this was only true when there were no results)
  • fixed an issue where invalid typed 'system:' tags were not being correctly cleaned
  • improved tag cleaning code
  • fixed fetching autocomplete results for small and specific searches (e.g. putting in 'a' will return results just for 'a'
  • fixed small and specific searches to return namespaced instances as well (e.g. putting in '8' can now return 'chapter:8'
  • fixed an issue with showing collected media in the preview window
  • fixed the file import status button not laying out correctly on edit subscription dialog
  • cleaned up some focus checking code
  • harmonised all initial key event processing
  • cleaned some key event processing
  • adjusted some server daemon timings to prefer post-boot update generation catchup over db maintenance
  • corrected a bunch of slightly mistaken unit test code
  • added some 'like' and 'numerical' ratings search unit tests
  • the advanced content update dialog now uses the new sizing system (so the go! button should no longer wander out of frame!)
  • reduced what I believe is excess fuzzy padding on the new sizing system
  • optimised a common client db request, speeding up a number of tag operations
  • gui now asks for confirmation when you make to delete a gui session
  • improved some out of date unit tests

• version 249

  • reintroduced shape and colour options to edit ratings service panels
  • reintroduced num_stars and allow_zero options to edit numerical ratings service panels
  • the export phrase--defaulting to '{hash}'--will now persist through export dialogs. it is saved whenever you click update or export
  • removed the unintended 'counts' that were appearing after related tags
  • fixed 'remove' action in custom filters
  • hitting the delete key on manage tags taglist now will always remove
  • added a BUGFIX option to not verify regular https traffic on the old networking engine for those non-Windows users who are getting SSL verify errors
  • refactored the top hover frame more, making it more flexible
  • created a new top hover frame for the duplicates filter
  • duplicates filter now also supports tag and ratings hover frames
  • duplicates filter now reports A or B as file index
  • added some placeholder buttons to the duplicates filter for actions and 'cog' customisation
  • reduced duplicate search shutdown logspam
  • improved duplicate search job cleanup
  • improved mime detection of thumbnail regeneration
  • improved some shutdown error handling
  • improved numerical rating search accuracy
  • improved some hdd import error handling
  • improved some unicode error handling
  • improved reliability of some database transaction processing
  • fixed animationbar having problems with single-frame videos
  • fixed namespace colour taglist not updating colour correctly
  • stopped the namespace colour taglist from deleting default namespace colours
  • made the new class of button more compact--not sure if I like it
  • refactored some db initialisation to avoid future transaction problems during special updates
  • tidied up some last incomplete taglist code from last week
  • did a little prep for some future shortucts overhaul
  • did some more menu code updating
  • added a simple subscription save/load unit test
  • moved outdated server unit tests forward
  • fleshed out some server certificate generation
  • some other misc v245 catchup work
  • cleaned up some pydeadobjecterrors caused by downloaders reporting progress to destroyed windows

• version 248

  • fixed two more issues with recent update code!
  • re-applied a more rigorously tested 'invalid subtag' replacement for any user that has experienced incomplete update code in the past few weeks

  • added some tag rendering options to options->tags that affect most user-facing, non-editing windows

  • added custom namespace-subtag connector (i.e. instead of ':')
  • added 'show namespaces' checkbox (i.e. to keep the colour but lose the namespace, like how most boorus display tags)
  • created a basic framework for the duplicate filter--it doesn't do anything yet, but you can launch it from the duplicates page to check it out
  • updated 'reset processing cache' action and added it to review services repository pages
  • review services now always starts on 'my files'
  • review services will try to stay looking at the currently selected service through a refresh_services event
  • fixed some thread-gui refresh interaction in review services
  • all listctrl-attached delete/remove buttons will now ask if you are sure before they go
  • reenabled db->maintenance->clear orphans, which it turns out I had already fixed by accident
  • serverside services will lock themselves more efficiently while generating updates, ensuring 'server busy' responses can always return promptly
  • fixed tag manage parents a/c entry to remove tags when they already exist in the box (previously, the a/c would only ever add, so a double-click was required to remove)
  • tuned adminside petition processing so it approves/denies in reasonable-sized chunks
  • refactored all the taglist gui code to focus on the underlying tag rather than the string representation
  • plenty of misc taglist cleanup besides
  • started a unit test suite for listboxes and wrote a thorough test for namespace colour listbox
  • misc canvas refactoring

• version 247

  • fixed a problem with deleting more than 256 files at once
  • furthermore, deleting from thumbnail view or after a filter will split delete jobs into chunks of 64 files at a time to reduce gui hang from deleting many tag-heavy files
  • rewrote canvas media container code to recycle containers, embed windows, animation bars, static image windows, and animation windows. scrolling through all kinds of media is less flickery (less 'grey box' window initialise flicker) and scrolling through static images should be completely flickerless! William Gibson slideshow speed works again!
  • this flickerless static image transition will be particularly useful in the forthcoming duplicate image filter!
  • got adminside petition processing working again
  • petition counts and fetching is now split by content_type and status
  • the approve/deny colour hint is more obvious on the petition panel
  • petitions now process off the main gui thread and throw up a popup message
  • added 'check all' and 'check none' buttons to petition panel
  • several serverside petition processing fixes
  • generalised and improved dynamic menu check item initialisation and inversion support
  • moved the 'get tags even if file already in db' option into a cog button on regular downloader pages
  • added a default menu option for 'get tags even if file already in db' to the same cog button
  • added this cog button to the edit subscription panel as well
  • fixed exporting tags to Hydrus Tag Archives
  • fixed exporting 'all known tags' to HTAs
  • cleaned some HTA and related code
  • fixed namespace based tag censorship
  • fixed autocomplete not filtering out current/pending counts if they are set to 'excluded'
  • fixed generation of non-expiring new accounts
  • fixed some v245->v246 tag improvement code that was replacing invalid tags with the incorrect namespace
  • patched a problem with '-:' dirty tag in the v245->v246 update code--I'm not sure what it was doing, but it catches the unusual problem and puts it in the 'invalid tags' category, so let me know if you get trouble with this in future
  • wrote unit tests for bytesdictionary and shortcuts serialisable objects
  • harmonised and improved how separators are appended to menus
  • cleaned up some client db index creation
  • cleaned up some client tuple stripping
  • misc pylint warnings cleanup
  • misc fixes

• version 246

  • fixed a critical bug in serverside content deserialisating that meant servers were not processing most client-submitted data properly
  • fixed a critical bug in 'all known tags' autocomplete regeneration--please run database->regen->a/c cache when it is convenient and let me know if your numbers are still off
  • fixed the pre-v238 update problem for good by abandoning update attempts and advising users to try v238 first
  • clientside invalid tags will now be collapsed like with the server last week. if a tag is invalid (typically something with an extra space, like "series: blah"), the update code will attempt to replace existing mappings with the collapsed valid version. some unusual cases may remain--they will be replaced with 'invalid namespace "series "' and similar. Please remove and replace these when convenient and contact me if there are way too many for you to deal with
  • duplicates pages now have a file domain for the filtering section, and they remember this domain through session loads
  • this file domain is accurate--counting potential duplicates and fetching pairs for 'show some pairs' only from those domains. the issue of remote files appearing should be gone!
  • there is now only one 'idle' entry in the duplicates page cog menu--it combines the three previous into one
  • fixed numerous irregularities across the wildcard code. all search input now has an implicit '*' on the end unless you put a '*' anywhere else, in which case it acts exactly as you enter it, with a non-* beginning matching beginning of string, whitespace, or colon, and non-* end matching end of string or whitespace
  • autocomplete now searches namespace, so entering 'char' will load up all the 'character:' tags along with 'series:di gi charat'. this can lag to hell and back, so it may either need some work or be optional in the future. feedback would be appreciated
  • typing 'namespace:' will include all the series tags below the special optimised 'namespace:*anything*' tag
  • autocomplete searches recognise an explicit '*' no matter where it is in the entry text. typing 'a*' will load up all the a tags and present a 'a*' wildcard option
  • quickly entering a wildcard entry will now submit the correct wildcard predicate ('rather than a literal 'hel*' or whatever tag)
  • review services panel now reports total mappings info on tag services
  • review services panel now reports total files info on file services
  • manage services's listctrl is now type | name | deletable and initially sorts by type. the strings used for hydrus service types are also improved
  • manage serverside services (called by server admins to manage their services) have fixed setnondupe port and name on edit service events
  • new popup messages will now also appear if there were previously no popup messages to display if the current focus is on a child on_top frame, such as review services (you'll now see the processing popup appear when you click 'process now' on review services)
  • the popup message manager now initialises its display window with a single message that is quickly dismissed. this helps set up some variables in a safe environment so they don't have to be generated later when the gui might be minimised or otherwise unusual
  • hid hydrus update files from 'all local files' searches
  • added 'media_view' entries for hydrus update files, just in case they are still visible in some unusual contexts (and they may be again in a future update anyway)
  • fixed 'recent tags' being returned from the database out of order
  • by default, 'recent tags' is now on for new users
  • 'get tags even if file already in db' now defaults to False
  • file import status now allows a 'delete' action below the 'skip' action
  • file import status right-click event processing is more sane
  • fixed the new raw width/height sort choices, which were accidentally swapped
  • cleaned the media sort code generally
  • cleared out some redundant rows that are in some users' client_files_locations
  • namespaced predicates are no longer count-merged with their namespaceless versions in 'write' autocomplete dropdowns
  • 'unknown' accounts should now properly resync after clientside service change
  • improved how registration keys are checked serverside when fetching access keys
  • fixed a v244 update problem when unexected additional tag parent/sibling petitions rows exist
  • improved my free space test code and applied it to the old v243->v244 free space test (it'll now test free space on your temporary path and report problems appropriately)
  • to improve log privacy and cleanliness, and to make it easier to report profiles, db/pubsub profiles now write to a separate log file named appropriately and labelled with the process's start time
  • profiles are more concise and formatted a little neater
  • across the program, three-period ... ellipses are now replaced with their single character unicode … counterpart (except to the console, where any instance of the unicode ellipsis will now be converted back to ...)
  • cleaned up some log printing code
  • cleaned up some experimental static serialisation code, still thinking if I like it or not
  • started on some proper unit tests for hydrus serialisable objects
  • fixed and otherwise updated a heap of unit test code to account for the v245 changes
  • cleaned up a bunch of old database table join code
  • started some databse-query-tuple-stripping code cleaning
  • deleted more old unused code
  • misc timing improvements
  • misc code cleanup experimentation
  • misc cleanup

• version 245

  • fixed a v244 update problem for clients updating from < v238

  • some misc stuff

  • if you start editing many subscriptions, cancelling a single dialog will break the chain of loading new dialogs
  • reduced some redundancy in regular client file import
  • improved how the dialog for selection from a list of strings works
  • if the client ever merges one directory to another (such as in external file locations migration) and any files fail to merge, the source will no longer be deleted
  • created new flexible bandwidth tracking and ruling objects
  • updated how repository updates work, splitting the old explicit and self-contained content update package system to a new implicit definitions/content split system and an improved one-step-sync metadata propagation
  • updating now takes approximately 23% as much bandwidth as before
  • update files are now stored in client_files and server_files like any other file (client_updates and server_updates folders will be deleted on update)
  • the server will print update generation info to its log
  • unfortunately, updates cannot be converted, so a complete resync of update files is required. the smaller update size and better bandwidth controls should mitigate the problem somewhat
  • the server has been compacted across all content types--its mappings db file should shrink about 12%
  • due to new service-specific ids, server.master.db should increase in size, typically about 50%
  • harmonised how GET and POST/response args are built and parsed across the network
  • server administration initialisation is now simpler, done with 'init' registration key
  • leading and trailing spaces are now removed from both the namespace and subtag components of namespaced tags, meaning 'title: blah' will be collapsed to 'title:blah'. tag repositories will clean their existing tags on update
  • invalid serverside tags will be replaced with valid placeholders on server update
  • refactored, harmonised and simplified some server request code
  • Eris makes a triumphant return to the root welcome page ('/' request of any service) with improved self-description text as well
  • updated how bandwidth is tracked and overseen
  • all requests now consume bandwidth
  • fewer requests are actually constrained by bandwidth--at the moment, it is only update files and file/thumbnail downloads, as these represent the overwhelming majority of bandwidth consumption and are not at all critical to service operation--this may change in the future, but it suits our purposes for now
  • bandwidth tracking code is more sane across the board
  • the server's administration service now tracks server-wide bandwidth as it happens
  • service and server-wide bandwidth rules are consulted as soon as the request begins
  • improved some server response rendering
  • permissions are more flexible and content-specific
  • improved how file repositories check and process file requests
  • created a gui control for managing bandwidth rules
  • updated the serverside service object
  • the serverside service object now contains bandwidth rules and tracking
  • updated how the server deals with services on a db level
  • refactored and cleaned a ton of server db code
  • updated how services are edited over the network
  • updated how the server associates its services with its http pipeline
  • converted clientside server service management gui to the new panel system
  • updated gui code for clientside server service management
  • converted clientside server service management db code to the new system
  • improved how bandwidth errors are reported
  • updated the account type object
  • the account type object now contains bandwidth rules and a more flexible permissions system
  • updated how the server deals with account types on a db level
  • converted clientside server account type management gui to the new panel system
  • updated gui code for clientside server account type management
  • serverside account types are now fetched from a cache, reducing memory sprawl
  • updated the account object
  • the account object now contains a bandwidth tracker
  • updated how the server deals with accounts on a db level
  • updated the clientside server object
  • the clientside server object now contains bandwidth rules and a tracker and has improved error management, recovery, and reporting
  • updated how the client deals with services on a db level
  • improved how serverside bandwidth errors are caught
  • all clientside services start with a 50MB/day bandwidth limit
  • existing repositories will get a 250MB/month, 50MB/day default bandwidth limit, just to help us get over the hump--see the release post for info on how to get all the updates anyway
  • updated manage services dialog extensively
  • service account registration now occurs through a simple button on the normal clientside service edit panel
  • all services can now be renamed
  • updated the content processing pipeline
  • prepared code for future merging of file and tag repositories
  • added future support for pend-petitions for files and mappings and simple creation permissions for tag siblings and parents
  • the clientside content processing pipeline now operates inside a single database transaction, reducing a great deal of previously redundant hard drive activity
  • the serverside content processing pipeline now creates per-service timestamped content definitions
  • the clientside content processing pipeline now maintains and appropriately consults a cache for server definitions
  • pre-process disk cache is more intelligent
  • updated review services to match the new service objects
  • service review panels now show more information about error state and so on
  • service review panels should now update as soon as services change
  • you can now force sync repositories from review services
  • you can now pause/resume repositories from review services
  • you can now export repository update files from review services
  • you can now import repository update files from the services menu
  • repository thumbnail download now uses a cache for faster thumbnail ownership testing
  • culled a lot of old code and experiments
  • deleted all the old messaging depot code and db table cruft
  • removed optimised petition processing
  • serverside deleteorphans is temporarily disabled
  • hydrus client-repository relationship no longer supports news
  • removed the 'stats' admin service call--it will come back as a account review page
  • clientside clear orphans is temporarily disabled
  • clientside local file/thumbnail server is disabled for now
  • custom service 'messages' for the root page are no longer available
  • some things are not working yet--they will be back in soon
  • content presentation on review services
  • service reset, other advanced content service controls
  • ipfs controls on review services
  • local booru controls on review services
  • most service-specific panels on manage services
  • petition resolution
  • serverside account modification, including banning

• version 244

  • updated client database to compact ( namespace_id, tag_id ) pair into a single id for storage
  • added some bells and whistles to the update code
  • added a free space check and messagebox warning before the update
  • updated db, service, and a/c cache creation code to reflect new schema
  • updated absolutely everything else in the db to reflect the new schema
  • for users with plenty of tags, the db should now be about 33% smaller!
  • unified how unnamespaced tag searching counts are totalled
  • unnamespaced tag searching counts are now totalled when the tags are fetched from the in-view ui media
  • unified how tags are split into ( namespace, subtag ) across the program
  • fixed deviantart gallery thumbnail parser
  • fixed linux session load page key event handling bug
  • os x can now support notebooks with zero pages open
  • fixed an issue where os x was losing the first page of some session loads
  • fixed some similar files shutdown work false positive calculation
  • reduced server bandwidth check period from 24 hours to 1 hour
  • improved calltothread scheduling under heavy load
  • improved scheduling of how files are physically deleted
  • numerous laggy temp_table replacement/cleanup
  • more temp_table replacement
  • misc efficiency improvements and general db code cleanup
  • misc path code cleanup

• version 243

  • updated more menu code to the new system
  • finished updating all main gui menubar and thumbnail menu code
  • all brackets, braces, parentheses, and single and double quotes are now ignored when matching search text to tag. inputting 'mercy' matches 'character:angela "mercy" ziegler' and entering '[[[intensifies]]]' matches '[intensifies]'
  • '*' searches will now return no autocomplete results, rather than trying to load absolutely everything
  • made wildcard searching significantly more permissable--it now implicitly applies a '*' to the beginning as well as the end of your search input
  • the main gui's notebook pages will now be disabled while a session loads
  • might have fixed a first-boot layout issue--otherwise added a layout entry to the debug menu for further testing of this problem
  • improved progress layout on file import dialog
  • created a more efficient way for threads to update gui elements
  • the file import dialog uses this new update system and parses lots of small files much faster as a result
  • the edit import folder dialog will now complain if you try to put in an import path that includes the install or db directory
  • ffmpeg mime parsing now catches and reports audio formats without any hassle
  • thread downloader no longer fetches file extension in filename tag 'filename:blah' instead of 'filename:blah.jpg'
  • changed png default zoom rules to copy jpg--the unusual zoom as default was more confusing than helpful
  • updated sqlite for windows--should be a decent bit faster
  • deleted a lot of old code that'll never be needed again
  • misc refactoring
  • misc cleanup

• version 242

  • optimised 'exact match' similar file queries to run a lot faster
  • optimised similar file queries in general, particularly for larger cycle queries
  • optimised hamming distance calculation, decreasing time by roughly 45%!
  • the similar files tree maintenance idle job will not trigger while there are phashes still to regenerate (this was redundantly and annoyingly blatting the new dupes page as soon as phash regen was paused)
  • removed similar files tree maintenance entry from db->maintain menu, as it can be done better from the new dupes page
  • adjusted the duplicate search and file phash regen progress gauges to reflect total number of files in cache, not the current batch job
  • all maintenance jobs on the duplicates search page will now save their progress (and free up a hanging gui) every 30 seconds
  • the duplicates page's cog menu button now lets you put phash regen and tree rebalancing on the normal idle routine, defaulting both to off
  • the cog menu can also put duplicate searching on idle time!
  • added a very rough 'just show me some pairs!' button to the dupe page--it is pretty neat to finally see what is going on
  • I may have reduced the memory use explosion some users are getting during file phash regen maintenance
  • wrote an unclose_page action and added it to the shortcuts options panel--it undoes the last page close, if one exists. ctrl+u will be the default for new users, but existing users have to add it under options
  • added ascending/descending sort choices for width, height, ratio, and num_pixels
  • the client can no longer talk to old http hydrus network servers--everything is now https
  • in prep for a later network version update, the client now supports gzipped network strings (which compress json a lot better than the old lz4 compression)
  • fixed gif rendering in the Windows build--I forgot to update a build script dll patch for the new version of opencv
  • the export file dialog's neighbouring .txt taglist file stuff now allows you to select a specific combination of tag services
  • if an hdd import's original file is due to be deleted, any existing neighbouring taglist .txt file will now also be deleted
  • the inter-thread messaging system has a new simple way of reporting download progress on an url
  • the handful of things that create a downloading popup (like the youtube downloader) now use this new download reporting system
  • sankaku seems to be 503-broke due to cloudflare protection--I have paused all existing sankaku subscriptions and removed the sankaku entry for new users (pending a future fix on my or their end)
  • I've also removed danbooru for new users for now--someone can fix the long-running sample size file issue in the new downloader engine
  • removed unnamespaced tag support from the hentai-foundry parser--maybe someone can try to fix that mess in the new downloader engine
  • menubuttons can now handle boolean check menu items that are tied straight into hydrus's options
  • menus launched from the newer frame and dialog code will now correctly display their help text on the main gui frame's statusbar! (at least on Windows! Linux and OS X remain borked!)
  • fixed a unicode error parsing bug in the gallery downloader
  • the server stop (or restart) command now correctly uses https!
  • the server test code now works on https as appropriate
  • fixed some misc server test code
  • misc fixes
  • misc cleanup
  • misc layout cleanup

• version 241

  • fixed the 'setnondupename' problem that was affecting 'add' actions on manage subscriptions, scripts, and import/export folders
  • added some more tests to catch this problem automatically in future
  • cleaned up some similar files phash regeneration logic
  • cleaned up similar files maintenance code to deal with the new duplicates page
  • wrote a similar files duplicate pair search maintenance routine
  • activated file phash regen button on the new duplicates page
  • activated branch rebalancing button on the new duplicates page
  • activated duplicate search button on the new duplicates page
  • search distance on the new duplicates page is now remembered between sessions
  • improved the phash algorithm to use median instead of mean--it now gives fewer apparent false positives and negatives, but I think it may also be stricter in general
  • the duplicate system now discards phashes for blank, flat colour images (this will be more useful when I reintroduce dupe checking for animations, which often start with a black frame)
  • misc phash code cleanup
  • all local jpegs and pngs will be scheduled for phash regeneration on update as their current phashes are legacies of several older versions of the algorithm
  • debuted a cog menu button on the new duplicates page to refresh the page and reset found potential duplicate pairs--this cog should be making appearances elsewhere to add settings and reduce excess buttons
  • improved some search logic that was refreshing too much info on an 'include current/pending tags' button press
  • fixed pixiv login--for now!
  • system:dimensions now catches an enter key event and passes it to the correct ok button, rather than always num_pixels
  • fixed some bad http->https conversion when uploading files to file repo
  • folder deletion will try to deal better with read-only nested files
  • tag parent uploads will now go one at a time (rather than up to 100 as before) to reduce commit lag
  • updated to python 2.7.13 for windows
  • updated to OpenCV 3.2 for windows--this new version does not crash with the same files that 3.1 does, so I recommend windows users turn off 'load images with pil' under options->media if they have it set
  • I think I improved some unicode error handling
  • added LICENSE_PATH and harmonised various instances of default db dir creation to DEFAULT_DB_DIR, both in HydrusConstants
  • misc code cleanup and bitmap button cleanup

• version 240

  • improved how the client analyzes itself, reducing maintenance latency and also overall cpu usage. syncing a big repo will no longer introduce lingering large lag, and huge analyze jobs will run significantly less frequently
  • the analyze cache will be reset on update, so you will have one big round of analyze the next time you maintain, and then you are good
  • added data structures to support auto-discovery of duplicate files
  • improved how some similar files maintenance occurs
  • flushed out duplicate status reporting
  • added a new page type, currently under pages->search pages->duplicates, to handle duplicate discovery and filtering
  • created a gui skeleton for the new duplicates page
  • started some handles and update code for the new duplicates page
  • wrote a new txt file in the db dir about the new emergency extract scripts
  • wrote an emergency extract script to migrate subscriptions to a new db
  • wrote an emergency extract script to migrate options to a new db
  • the trash clearing daemon now runs in the foreground, and a little of its code is improved
  • the trash clearing daemon now makes a popup message when it does work
  • the server's ssl keys are now set to read-only on Windows and user read-only only (i.e. chmod 400) on Linux and OS X on creation and update
  • added a explicitly unicode popup message to the debug test
  • fixed some network error catching code that was using Windows-only error codes
  • converted more of the thumbnail right-click menu over to the new system
  • improved some listctrl code
  • misc cleanup

• version 239

  • finished up similar files search data maintenance code
  • similar files search data maintenance will now run during idle time
  • similar files search data maintenance can be called from database->maintain menu
  • the crowded database->maintenance menu is now split into maintain, regenerate, check
  • improved the similar files tree generation code, speeding searches significantly
  • wrote a new listctrl class to handle more complicated objects and also sort by underlying data
  • the new listctrl now handles object name non-duplication
  • cleaned a bunch of crap old listctrl code
  • manage export folders now uses the new listctrl
  • manage import folders now uses the new listctrl
  • manage subs now uses the new listctrl
  • manage scripts now uses the new listctrl
  • options media viewer options now uses the new listctrl
  • file import status panel now uses the new listctrl
  • the new listctrl can now quickly fetch item index from the underlying object
  • the file import status panel should now cope with extremely huge lists a bit better now!
  • multiple parsing child nodes can now import from/export to clipboard as lists
  • export folders now have names, so you can have multiple export folders pointing to the same path! existing export folders will get their path as their name, but this can be changed no prob
  • cleaned import/export folder dialogs
  • cleaned import/export folder dialog workflow
  • several misc import/export folder improvements
  • hydrus servers are now exclusively https with self-signed certificates
  • hydrus servers now create server.crt and server.key in their db folders for SSL--these files will be backed up along with everything else on an admin backup command
  • system:hash now ignores the file domain and any other predicate. it now returns very quickly, no matter the context
  • improved system:hash search logic
  • all the awkward choice dropdowns in system predicate panels are replaced with radioboxes
  • improved system:rating panel grid layout
  • wrote a better subclass of radiobox to handle more data
  • moved first half of thumbnail menu to new menu system
  • cleaned up a little thumbnail menu logic
  • improved the different ways services are added to thumbnail menu
  • thumbnail 'select' menu is logically cleaned up and allows for better file domain selection
  • the thumbnail menu's copy files and copy hashes to clipboard will now send them ordered (they were previously pseudo-random)
  • added 'paths' to the share->copy thumbnail menu for copying multiple files' paths. these are also ordered.
  • if the popup message manager does not have any errors, it will no longer unhide (which can annoyingly raise the main gui window) when the gui window does not have focus
  • removed some old redundant error reporting stuff in popups
  • improved and quietened some some mime detection failure code, sped up mime failure loop in all cases
  • massively simplified and atomised how new serialisable-object management panels can save their data
  • the manage subscription dialog now saves in a single, faster transaction
  • the manage script dialog now saves in a single, faster transaction
  • reduced redundant index work from analyze jobs
  • improved tag parse error handling
  • fixed media removal rules when deleting from the 'all local files' domain
  • polished and clarified some of the help's tag schema
  • misc cleanup
  • more misc cleanup
  • misc refactoring
  • more misc refactoring

• version 238

  • added left/right equal ratio preference to the similar files search tree generation
  • added high standard deviation preference to the similar files search tree generation
  • added 'db profile mode' to the help menu, for briefer profiling in future
  • when a similar files search occurs with db report mode on, it'll popup a nice bit of text saying how many cycles it took
  • fixed a bug due to last week's 'all local files' change that meant archive/delete filtering would include any selected trashed files
  • 'recent tags' will now only be populated by add/pend tag events, and only by those tags that were valid for that action
  • moved the help menu to the new menu system
  • added bitmapped menu item support to the new menu code
  • fixed and cleaned up some of the new menu code
  • improved a/c cache regen error reporting

• version 237

  • added 'all local files' service that spans all local file domains
  • improved trash service code
  • trashed files now report their trashed timestamp in right-click menus
  • trash views will sort oldest/newest by the trash timestamp
  • renamed 'local files' to 'my files' to reduce initial confusion
  • if you don't like 'my files', you can now rename the local files service under manage services!
  • improved how some local file service metadata is stored
  • cleaned and possibly fixed up some delete code
  • cleaned up a bunch of misc service and file service code
  • the client is more intelligent about what local files are -- you can now 'open externally' trashed files, for instance
  • on multiple monitor systems, the new sizing system now bounds itself by the appropriate monitor's dimensions (previously, it was always consulting the primary, I think)
  • if an expansion event causes a frame to grow off screen, the new sizing system will now attempt to move it up and left so it is completely visible
  • fixed an important bug in the specific service autocomplete cache that was leading to several kinds of miscount--please regen your autocomplete cache at your convenience
  • the client can now post additional messages on boot (you'll see one!)
  • improved how errors with unusual characters are applied to failed import file status objects
  • left double-click on the main gui greyspace now opens the new page chooser as well
  • restored session pages now recover more gracefully from missing services
  • shortened the 'stop after this many files' phrase, which was mis-sizing downloader panels
  • changed several bits of old jank in the 'import options - files' collapsible. it is also now thinner
  • misc gui improvements
  • updated more of the menu system code
  • the selection tags box now sizes its height more reasonably
  • fixed 'check now' and 'reset cache' buttons on edit sub dialog
  • subscriptions report better file import status on their popup
  • misc cleanup

• version 236

  • in prep for a network https upgrade, the client can now detect and escalate to https when making connections to hydrus services
  • import/export to png and clipboard now supports multiple objects at once!
  • rewrote the manage subscriptions dialog to work on the new panel system
  • the new manage subscriptions dialog has a listctrl and a sub edit dialog
  • the new manage subscriptions dialog has the same add/export/import/dupe/edit/delete buttons as the manage scripts dialog
  • subscriptions are now importable/exportable, including en masse with the new multiple object import/export support!
  • the new manage subscriptions dialog has retry failed/pause-resume/check now/reset buttons for easy mass subs management
  • the edit subscription panel has a bit of a layout makeover
  • the edit subscription panel now updates itself as its buttons are hit
  • the edit subscription panel disables buttons that are not applicable
  • subscriptions can now be renamed!
  • cleaned some misc subscription code
  • relabelled initial and periodic file limit in the subscription edit panel
  • middle-clicking on the main gui's greyspace (e.g. to the right of the notebook tabs) will spawn the new page chooser!
  • created a simple HydrusRatingArchive class--will do more with it in future
  • added ffmpeg, python, and sqlite versions to the help->about window
  • harmonised daemon code
  • added a new class of daemon that will not fire while a session load is occuring
  • subscriptions, import and export folders, and file repo downloads now use this new daemon
  • cleaned the way background daemons check for idle
  • expand/collapse panels now notify the new kind of toplevelwindow that a resize may be needed when they switch state
  • time deltas (like on subs edit panel or a thread watcher) now render more concisely ('7 days' instead of '7 days 0 hours')
  • serialisable object png export panel now has a width parameter
  • fixed a bug where tags that begin with unicode digits were accidentally identifying as numbers for the purposes of sorting and throwing errors on convert fail
  • the media viewer can handle some more unusual content update combinations--for instance, if it cannot figure out which media to show next, it will revert back to the first image rather than displaying an undefined null mess
  • updated and cleaned a bunch of my old misc encryption code
  • misc cleanup

• version 235

  • finished first version of new faster dupe search--'system:similar to' now uses it
  • finished faster dupe search tree creation
  • finished faster dupe search tree search
  • added faster dupe search tree leaf insertion
  • added faster dupe search tree rebalancing maintenance
  • added faster dupe search tree orphan deletion maintenance
  • added faster dupe search leaf-regeneration-as-scheduled maintenance
  • misc faster dupe search tree work
  • new search tree will be created on db update
  • fixed the 'add' buttons on the import files dialog
  • fixed linux manage tags text input
  • reduced incidence of linux media viewer post-manage tags key event swallowing
  • fixed an issue where if the manage tags dialog was launched from the media viewer while a hover window was displaying, the main gui would often be raised when the manage tags frame was closed, annoyingly obscuring and de-focusing the underlying media viewer
  • fixed manage tags action workflow when no valid tag choices exist
  • entering a tag petition reason now has several common 'suggestions' for quick entry
  • removed ffmpeg binary from default linux release (as basically every linux has a decent ffmpeg by default)
  • fixed the incorrect ffmpeg version for os x
  • added an option to options->media to prefer to use the system ffmpeg over any in the hydrus bin directory
  • fixed linux arrow key navigation of the page picker dialog
  • cleaned up the page picker dialog's gross old code
  • the edit file lookup script panel now has a script management control on its test page to better report on and control tests
  • scripts will new correctly attempt to decode unicode and other encoded text in responses
  • if non-text data is accidentally fetched in the file lookup script panel's test page, it is now caught and a small summary of that event is printed to the text box instead
  • added the same non-text catch to the edit link node panel
  • improved how some script errors and status updates are handled so the new test window script management controls will see and report more useful information
  • removed 'restart' option from linux for now--looks like a bug in pyinstaller
  • improved how the new GET request read reattempt works
  • cleaned up some spammy error code
  • cleaned up spammy test code
  • misc error handling improvements
  • misc cleanup

• version 234

  • the action choice workflow in the manage tags dialog now merges decisions for multiple tag entry events
  • the button dialog that pops up on the new merge manage tags workflow has button tooltips to better describe the proposed action's tag and file combinations
  • the manage tags dialog only requests one petition reason for an above merged multitag petition event
  • perceptual hashes are now stored in the cache db (moved from the preferably leaner main db)
  • the database now supports multiple perceptual hashes per file
  • fleshed out the perceptual hash vptree generation and maintenance code
  • added option for main gui title to options->gui--it even updates live
  • added unicode path support when importing serialised pngs
  • added unicode path support when exporting serialised pngs
  • exporting a serialised png now reports success via the export button, which will temporarily relabel itself
  • added discord links to help files and help->links menu
  • when a GET network connection fails during the read phase due to an unexpected timeout, the request will be reattempted a couple of times, like failed connection initialisations currently are
  • escape key now closes scrolledpanelframes (review services or import status frames)
  • manage tags dialog and frame now closes due to escape key correctly
  • fixed a size calculation bug that was not initially drawing scrollbars on manage options and any other listbook-containing scrolling panel when the screen is too small to show the whole dialog
  • undo menu now works on the new menu system
  • cleaned up some bad gui-thread interaction in the file import dialog
  • fixed file->restart in the built release, including when the install path includes a space
  • the trash service no longer records which files it has physically deleted, as this information is not used and is redundant compared to the existing local files' deleted record (existing records will be deleted on update)
  • the subscription daemon will wait 90 seconds after boot before triggering--quitting the client before then will result in subs not being checked

• version 233

  • made a plan for faster dupe search
  • created skeleton of db tables for faster dupe search
  • wrote out search algorithm for faster dupe search when the rest is ready
  • updated search algorithm and skeleton to support a future multiframe (i.e. gif/video) similar files comparison
  • the os x release now has 'client' as the main client executable
  • the os x release now includes the server, under the 'server' executable
  • added studio (128,0,0) and meta (0,0,0) default namespace colours
  • the 'add all' button on the file lookup tag suggestions panel now will only ever add--it won't remove/rescind pend
  • the password system now supports non-ascii input (be careful though, as even a subtle change in keyboard encoding that nonetheless may have the same visual characters will likely be considered a different password)
  • fixed the 'nonetype has no dtype' rendering problem introduced by last week's 16-bit channel fix (this mostly affected static gifs)
  • fixed non-null ratings changes in the numerical ratings dialog
  • fixed culling and adding variables not initialising on the edit html formula panel
  • fixed the htmlparser attribute fetcher to deal with both single value attrs ('id') and multiple value attrs ('class'). in the latter case, the many values will be joined up, as how they appear in html
  • tweaked disk cache timings a little more to account for more scenarios--it now also reports itself to the shutdown splash screen

• version 232

  • finished png object sharing system, created some gui for it
  • switched the top half of the new png export to wx code, which can render with prettier system fonts
  • you can now 'export' parsing scripts to png from manage parsing scripts for easy sharing
  • you can also 'import' parsing script pngs in the same dialog!
  • the file lookup tag suggestions panel now listens for media update events (so if you browse the media viewer with manage tags open, it'll keep up with the current file)
  • added a 'add all' button to quickly add all tags from file lookup suggested tags panel
  • created a 'script management' control for file lookup suggested tags panel and added some script text status updates for it (progress gauge doesn't do anything yet)
  • added basic 'cancel' support to script management control--mid-http cancel support will come with the gauges
  • added url support to thread notification objects, and hooked script code into it
  • added a url menu button to script management control, so you can launch any parsed urls the script found in your browser
  • html parsing formulas now support positive and negative front and end character culling
  • html parsing formulas now support prepending and appending arbitrary text
  • added some more pretty text to the html formula summary to explain when they do the new culling or adding
  • script 404s and other network errors are now caught neatly and reported
  • link node 404s and other network errors are now reported and recovered from
  • fixed favourite file lookup script selection on initialisation
  • fixed parsing veto test to include membership of search string in result, not just exact match
  • fixed html formula tag attribute fetching
  • updated os x release to python 2.7.10
  • fixed os x about window
  • fixed v231 linux release image rendering crash
  • updated linux release to wx 3.0.2.0
  • updated linux release to opencv 3.1.0
  • fixed rendering for 16-bit-per-channel images
  • fixed a potential out-of-order settagboxfocus event issue on panel initialisation
  • created a 'menubutton' control and switched existing code over
  • created some 'nullipotent panel' dialog code to handle some new stuff
  • misc fixes
  • bunch of misc refactoring

• version 231

  • added file lookup scripts suggested tags control and appropriate options panel
  • added final bit of gui->filelookup script->content update pipeline conversion and tie-in code
  • moved all the suggested tag options into a notebook
  • added notebook layout for suggested tag control columns. it is now the default
  • suggested tag columns now have a unified set width
  • the parsing ui's add script and node buttons now spawn menus rather than the awkward listofstrings dialog
  • added 'iqdb danbooru' file lookup script. it hangs the ui pretty bad atm, but that will change in future
  • fixed a critical pyinstaller problem with the os x builds
  • log files will now be appending with 'year-month', and will roll over to a new file as the month turns
  • help->about dialog now has some library version information
  • converted more menu stuff to the new system
  • wrote the guts of a new png-based object sharing system
  • dropdown choices are more resistant to missing init and invalid defaults
  • tweaked some disk cache maintenance timing
  • fixed a multi-version update bug regarding external thumbnail paths not being initalised
  • misc cleanup
  • misc cleanup, improvements

• version 230

  • all multiline text ctrls now support ctrl+a for select all, wew
  • added dynamic menu_item tracking to new menu id system
  • added submenu tracking to new menu id system
  • added comprehensive menu, submenu, and menu item destruction to new menu id system.
  • 'pending' menu now works on the new menu id system--is likely a major cause of the menu id bugs
  • added 'info' tabs to the script/nodes/formula parsing gui panels
  • added a 'veto' content type for discovering undesired redirects
  • added special veto support to parsing engine
  • expanded the content node panel to handle and display multiple content types
  • added 'link' parsing node panel
  • expanded 'link' panel test page and tied in secondary layer example url and data forwarding to children
  • fixed the parsing node children control not forwarding example data up to nodes on edit
  • file lookup url is now explicitly stored in the file lookup script object
  • file lookup url is now passed to child parsing dialogs for test example data purposes
  • expanded internal 'requests' interface a little--it now also always includes the 'user-agent' request header as the typical 'hydrus/version_number'
  • wrote script GET/POST engine gubbins and tied it into the respective test page
  • wrote link GET engine gubbins and tied it into the respective test page
  • 'person' namespace added to default namespace colours--it is RGB ( 0, 128, 0 ), a darker version of 'character' green

• version 229

  • edit html formula dialog can now be launched from edit content node panel
  • edit html tag rule dialog can now be launched from edit formula panel
  • edit formula panel test now works
  • edit content node panel test now works
  • edit script panel test now works
  • html formula now filters out missing content rather than complaining
  • misc parsing work
  • fixed some listctrl resize columns to be more helpful
  • attempting to size a resizing listctrl column will now move the resizing column to the last column, which will avoid the flickering scrollbar sperg-out of previous
  • attempts to resize the final column of a listctrl while it is the resizing column will be silently vetoed, which will avoid more resize event flickering
  • frame location options panel now allows for negative position coordinates (for multi-monitor setups)
  • all the new resizing panels now handle certain size updates in a more unified and reliable way
  • fixed an issue with hover windows' mouse position check breaking during media window init/shutdown edge cases
  • avc1 mp4 support added
  • ftypFACE mp4 support added
  • ftypdash mp4 support added
  • mpegts mpeg support added
  • to help migration, paths are now only stored as portable (i.e. relative to the base path) if they are beneath the base path (previously, this was true for any path on the same partition, permitting PITA '../../my_files' portable paths)
  • to help migration, portable paths are now relative to db_dir, not installation base_dir (existing client_files_locations, ideal client_files/thumbs_locations, export_path, and HTA paths will be updated)
  • started work on cleaning up button code across the program
  • created a better way to bind menu events that shouldn't suffer the id overflow bug
  • started work on replacing the old id_to_action_cache menu mess with the new system
  • too-many redirect exceptions are now richer and caught more sensibly
  • circular redirects are caught as soon as they occur
  • reduced sqlite cache size (with the new disk cache, this is now less important)
  • client.py is now included in the source and windows frozen releases

• version 228

  • added support for RIFF .avi files
  • fleshed out parsing ui, made progress on node editing, moved edit/test panels to notebook pages to reduce clutter
  • the video renderer now initialises off the gui thread, which should reduce some video browsing chunkiness
  • trying to save a session that would overwrite another (from the same name) now throws up a yes/no warning
  • reintroduced '--no_daemons' and '--no_wal' to the new command line argument parser
  • you can now set the default tag service for new search pages under the file->options->tags page
  • since it can now be anywhere, added 'db directory' to the file->open menu
  • fixed a build capitalisation issue that was making the windows frozen exe release crash when trying to run from source
  • the 'too many redirects' error message is expanded to print the url called and its redirect request
  • errors from trying to delete files in use by another process will no longer make popups--they'll just write a note in the log
  • improved how some errors are written to the log
  • misc fixes

• version 227

  • both the client and server now use standard command line input and will produce a proper help with a -h switch (although the 'windowed' frozen client executable can't print back to console)
  • both the client and server now support a -d or --db_dir to set a db directory outside of the install directory. you can now run multiple clients off the same install!
  • critical boot error reporting is more reliable
  • if the admin port is in use on server boot, the error is handled better and the program will quit more easily
  • fixed mime parsing for some unusual webms
  • added support for even more unusual webms
  • added support for SonyPSP mp4s
  • fixed system:hash when searching for a non-existent non-sha256 hash
  • fixed layout of the 'dismiss all' popup button
  • preview windows will no longer update in the background if the 'hide preview window' option is set
  • misc fixes
  • misc cleanup

• version 226

  • fleshed out a bunch of parsing script engine and gui framework and added an 'under construction' entry to the services menu. feel free to play around with it!
  • fixed autocomplete same-service sibling count merging
  • sped up how autocomplete counts are put together
  • results will build faster due to an improvement in how tag rows are fetched
  • results will build faster due to an improvement in how tag strings are fetched
  • incidence taglist sorting now secondary sorts by lexicographic in the 'correct' direction (a-z for desc, z-a for asc)
  • all listctrls now support ctrl+a to select all
  • cleaned up some superfluous and possibly buggy canvas zoom calc when using the open externally button
  • the zoom_switch event will now reset the drag coordinates (recentering the image) if the resultant zoom fits into the canvas frame
  • the mouse cursor hides itself more quickly on media drag on Windows (so warppointer jitter is hidden)
  • the popup message manager will be careful about switching from hide to show while the parent gui is minimised on windows, which should reduce the grey box problem
  • the popup message manager will specifically re-layout on a transition from hidden to shown to stop the 'crushed into a corner' layout problem that would sometimes happen here
  • popup message manager will now consider a minimsed gui as unfocused for the purposes the of the focus BUGFIX option
  • added a new delayed popup message item to the debug menu for testing minimised popup creation
  • vacuum maintenance period option is moved, reset to larger default of 30 days
  • 'last session' will now always be updated on client close, even if it is not the default session

• version 225

  • system:numtags is now much faster when applied to 'local files' or a file repository
  • system:numtags now correctly counts same-tag-different-namespace tags as distinct (for instance, [page:1, chapter:1] was previously being counted as only one)
  • refactored fast db integer iterable access code into a context manager
  • media result building is faster
  • added namespace-grouped incidental tag sorting
  • fixed the OpenCV image loader for monochrome images
  • fixed a thread interaction issue when drawing popup messages
  • popup messages should no longer flicker while static
  • popup debug test is richer
  • fixed editing of some taglists such as in explicit tags and import tag editing
  • SSL EOF errors are now caught by the networking engine

• version 224

  • rewrote the static image rendering and caching pipeline -- images are now resized on the fly, and only the master image is cached
  • all image rendering and zooming is faster and consumes less memory
  • image rendering now obeys the zoom quality options in the 'media' options page!
  • static image zooming will take advantage of OpenCL (video card acceleration) wherever available
  • added an option to the 'media' panel to allow fast but potentially unstable opencv image loading
  • fixed the tumblr gallery downloader (tumblr put an extra character in their API response, wew)
  • the debug code profiler now prints more information
  • cleaned and updated reducing lag help page to match new profiler
  • 'recent' suggested tags are now sorted by recency
  • the thread watcher now checks url history as well as md5 match to compensate for cloudflare optimisation making for unreliable api and causing dupe downloads
  • misc fixes

• version 223

  • fixed the popup message manager's bad minimise recovery
  • added an option to automatically 'hide' the popup message manager on main gui minimise (which has patchy multiplat support--some mindow managers don't do this automatically, some do but break when you force it)
  • added an option to automatically hide the popup message manager on main gui dofocus, which may help with window managers that minimise to the system tray, which wx cannot detect
  • added a 'make some popups' test job to the help->debug menu
  • siblings will now present in a more service-specific way
  • many gui elements that display tags are now aware of their tag service for the purpose of collapsing siblings
  • simplified a whole load of the siblings code
  • in some conditions, media will load faster
  • autocomplete tag censorship will now also apply on in a service-specific way in 'all known tags' queries
  • another remote connection reset exception is now properly caught by the network engine
  • the embed button will now draw the correct background colour behind transparent thumbnails
  • the animation scanbar will no longer scan if it has not previously experienced a mouse_down event--it could sometimes inherit this status from a previous media filter or embed reveal, resulting in undesired instant scan and undefined animation canvas behaviour
  • advanced content update panel will no longer list 'copy' as an action if 'local tags' is the only tag service
  • fixed a bug when performing a 'go!' action in the advanced content update panel when 'local tags' is the only tag service

• version 222

  • created a 'raw url' downloader page that just downloads urls and tries to import the result. it has a 'paste urls' button to make mass import of a list of urls easy
  • fixed an options update bug when updating to v221 from any version before v220
  • added support for 'ftypqt' quicktime (usually .mov) video
  • embed button now uses system gui colours
  • embed button now puts the thumbnail of the media, if one exists, behind the 'play' button
  • sped up an inefficient existing mapping check that was slowing new pending mappings for popular tags
  • 'last session' will no longer be listed on the gui session delete menu
  • cleaned up the main gui's initialisation events--a sizing bug often triggered after system reboot may be fixed
  • popup messages are initialised in a safer way
  • popup messages are dismissed in a safer way
  • popup messages will hide/restore themselves more reliably when the main gui window is minimised/restored
  • the pending popup message queue is now regularly purged of already deleted messages
  • new popup messages will no longer raise the main gui window to the top
  • subscription http errors during the gallery sync phase are now caught and handled gracefully, with exact error text written quietly to the log
  • network timeouts during successful response read are caught and converted to a hydrus network exception that will be caught and handled more reliably up the chain
  • the client's upnp daemon will now silence upnp mapping errors that are due to the router being too busy or full or any other unknown errors. a simple statement about the error and an instruction to explore the problem with the manual upnp manager will be written to the log
  • finished flexgrid refactoring
  • the new automatic flexgrid creation detects subsizers and lines them up more accurately with standard controls
  • wrapped the different sections of the 'speed and memory' options panel into staticboxes
  • wrapped the misc crap up top the 'tags' options panel into a neater staticbox
  • taglists with unusual tags will copy them more reliably and present fewer invalid menu options
  • misc layout fixes

• version 221

  • fixed the 8chan thread parser for the new sha256 file urls. legacy links should still work!
  • created a 'x recent tags' suggested tags control and all the db and so on to go with it
  • recent tags defaults to off--turn it on under options->tags
  • replacing the previously basic zoom_in_to_canvas, the zoom options now support 100%, max regular zoom, or canvas fit for scaling up and down and for the media and preview windows!
  • cleaned up some garbage zoom code
  • refactored and harmonised how default zooms are calculated
  • harmonised and improved how canvas zooms are set
  • zoom_switch now works for flash
  • if it looks like the last instance of the client did not shut down cleanly, a new client will now present a dialog to choose whether to attempt opening the default session or just a blank page.
  • all listctrls across the program have better secondary sort
  • added an isatty() stub to the HydrusLogger for any unusual error handlers wanting to call it (youtube parser pafy was doing this)
  • removed some accidental (123) counts from some predicate menu and clipboard rendering
  • refactored a bunch of redundant flexgrid creation code
  • did a bunch more flexgrid refactoring
  • refactored some ugly global variables
  • misc mvc cleanup
  • misc cleanup

• version 220

  • fixed collection selection show action lookup for the preview window
  • files added through the normal import dialog or import folders will be not be fed into the import pipeline if it looks like they are already in use by another process
  • (hence setting an import folder to a browser download destination should be more reliable)
  • rewrote some critical phash generation code that was running extremely slow when importing large pngs with transparency
  • all listctrls that have an edit button now also trigger the edit event on item activate (e.g. double-click/enter key on an item)
  • added zoom in/out/switch to edit shortcut action dropdowns, can't promise it works everywhere
  • 'stopping' a gallery page search will now correctly make sure the search queue is unpaused so it can move on to any subsequent search
  • the way excess messages are added to the popup message manager is quicker, improving gui responsivity on message spam
  • fixed a potential race condition when a file import occurs at the same time as client_files rebalancing
  • fixed a potential race condition when a check_file_integrity db maintenance occurs at the same time as client_files rebalancing
  • the client will no longer suppress some core debug stuff
  • fixed an invalid event handler on canvas close bug
  • fixed a long-time wx locale issue
  • fixed a misc main gui parent assert issue
  • fixed some invalid non-wx-thread calllater calls
  • cleaned up some inartful showmessage calls
  • cleaned up some datacache init code
  • fixed some shutdown event handling
  • generally improved and fixed in some cases how threads signal job status
  • improved some thread-interaction timings
  • improved how files and thumbnails are deleted
  • file imports add their files to the client_files structure in a more sensible way
  • misc fixes
  • misc layout fixes
  • misc cleanup

• version 219

  • wrote framework for per-mime zoom options--mimes now have separate show actions for the media viewer and the preview window and zoom in to fit
  • added 'half/double zoom' to limit a mime's zoom to only 25%, 50%, 100%, 200%, 400% and so on (png defaults to this)
  • added placeholders for zoom in/out algorithm quality, but they don't do anything yet
  • added a listctrl to edit all this to options->media
  • added a scrappy zoom values option to options->media
  • cleaned up a bunch of zoom code
  • added video/mpeg support
  • fixed some mime layout stuff
  • duplicate tag predicates across the 'all known tags' space are merged into more accurate (3-5)-type counts
  • refactored and simplified how predicate counts are merged
  • if launched from the manage tags dialog, the advanced content update dialog now correctly filters hta imports to only the files the manage tags dialog was launched with
  • the advanced content update dialog has a bit of better text to explain this
  • in prep for a complete rewrite of the image rendering pipeline, merged the fullscreen and preview image caches and cleaned up some related code
  • added resume recovery to the v215->v216 update code for users for whom this update was interrupted
  • directory creation is improved across the program
  • fixed the optimised merge-move file code, which was forgetting to clean up the source when no move was needed
  • the optmised merge-move directory code will now be much quicker in some situations
  • all file moves across the program should be less stupid-error prone, and repeating/resuming many maintenance or update tasks that require a lot of moves will be much faster
  • manual exports, which use copy rather than move, should also be more sensible and faster when repeated
  • improved database exception rendering, which in some cases was being truncated
  • misc cleanup
  • misc small updates to help

• version 218

  • related tags control now updates on new media events (i.e. if the parent media viewer scrolls to different media)
  • related tags also initialises with suggestions and refreshes the list on media change
  • related tags now properly filter out dupe suggestions caused by siblings
  • went over all the gui colours in the client, cleared some redundancies and replaced all the static whites and blacks with appropriate system defaults or chosen custom colours
  • full size thumbnails can now also be stored in a different location, like resized thumbs, under options->file storage locations
  • fixed false positive shutdown maintenance checking due to vacuums projected to take longer than the given maintenance time
  • fixed false positive shutdown maintenance checking due to sqlite_stat1 creeping in
  • pasting tags or otherwise editing many in at once in the manage tags frame will immediately commit all the changes in one db transaction, rather than one for each tag laggily in turn
  • removed 'import tags' from file menu--this advanced stuff is now done in and better left to the advanced content update dialog
  • restoring from an old db backup with a simpler db structure will no longer create conflicts
  • deleting currently rendering files from the trash should be slightly more reliable
  • fixed some crash error reporting
  • misc cleanup

• version 217

  • fixed some high-res video streaming thread scheduling problems with the new video renderer
  • fixed a cause of huge memory bloat with greatly upscaled videos
  • to improve seek response time, streaming buffer for the video renderer has a much smaller cap
  • renderer throttling calculations are more sensible and reliable
  • the video renderer discards frames to save time if they happen to still be in its buffer
  • the video scanbar now displays the current frame buffer around the caret!
  • video canvas now recycles the same frame blit bitmap to save a little time
  • wrote a prototype related-tags suggestion 'service' for the suggested tags control
  • you can turn it on and set some options for it at options->tags, feedback would be appreciated
  • munged increasingly complicated components of the suggested tags control into a clean and proper self-hiding panel
  • fixed a very important bug that was failing to filter visible thumbnail fetch on mass select and thus massively slowing down the client on large ctrl+a-like operations
  • open externally button now shows the media's thumbnail, if it has one
  • open externally and embed buttons now use hand cursor
  • the simple path tagging dialog panel now cuts off .jpg extensions from filenames on filename parse
  • if the string component of a generated file export path already ends in the correct .jpg extension, a second will not be added
  • ipfs unpin will no longer break if the file was already unpinned
  • the hydrus server now gives filename (for a file save as dialog) correctly on a content-disposition header (this affects the client's local booru as well)
  • the secondary sort can now be a namespace or rating sort
  • fixed some potential init problems with some dropdown controls
  • an edge case object-missing cache retrieval bug is fixed
  • updated openssl on os x, which might have fixed some problems
  • updated python on windows, which updated openssl and a bunch of other stuff
  • updated sqlite on windows
  • updated linux dev machine to ubuntu 16.04, so a variety of packaged libraries are updated
  • fixed auto server setup if the client is launched from a windows cmd window
  • misc cleanup

• version 216

  • video rendering pipeline rewritten to be much smoother
  • canvas video render timing rewritten to be more frame accurate and smoother
  • video rendering pipeline will deal with 100% CPU rendering bottlenecks much better
  • videocontainers will rush to render towards clicked cursor and then slow down
  • videocontainers will stop rendering as soon as they are replaced on zoom
  • videocontainers will stop rendering as soon as their parent canvas is closed
  • added a video buffer size option to options->speed and memory
  • client_files subfolders are now split into f, t, and r groups for files, full size thumbnails, and resized thumbnails
  • added an option to override resized thumbnail storage location under options->file storage locations
  • added secondary sort option to options->sort/collect
  • fixed a bug where automatically pended tags could exist on top of current tags, forming redundant (1) (+1) situations
  • fixed a bug when sorting by namespace and a respective namespaced tag has a sibling that de-namespaces it
  • relatedly, namespace sorting will now filter correctly by collapsed siblings when those sibling pairs' namespaces differ
  • and collecting will now collect correctly with sibling pairs where the pairs' namespaces differ
  • simplified how the server caches its account sessions
  • improved how the server refreshes specific account sessions
  • the server now refreshes cached session accounts immediately when an admin actions an account type
  • neatened a little server db account fetching code
  • fixed file repository petition denial
  • added a better catch for upnp addmapping error when the mapping already exists as a port forward
  • the media canvas will no longer accept any mouse or key event if a menu is open anywhere in the program (this circumstance is difficult to pull off, but does cause a crash)

• version 215

  • you can now set different suggested favourite tags for different tag services
  • added an option to tags option panel to set fixed suggested tags control width
  • file import status now works on the new window sizing system, and when launched from main gui uses file_import_status frame key
  • file import status is now a generic apply/cancel dialog when launched from import folder or subscription dialogs
  • these file import status dialogs are reenabled for linux and os x, as they work properly now!
  • very large images will render faster and more beautifully on a freshly launched fullscreen media window
  • all zooming in will be higher quality, particularly for sharper images
  • better zooming in code also works for very large animations/video
  • started a framework for better html parsing
  • updated sankaku downloader to support 'medium' namespace
  • analyze calls no longer analyze sqlite_stat1, which isn't needed and sometimes gives awkward maintenance timings
  • fixed server's deleteorphans, which wasn't working with the new thumbnail names
  • improved speed and weight of server's deleteorphans
  • fixed a missing foreign key in file_petitions
  • improved thread worker creation and job processing scheduling
  • fixed os x media viewer window float_on_parent status
  • default gui session now defaults to 'last session'
  • gui recovers from missing default session on boot more gracefully
  • crash log code improvements now applied to all boot scripts
  • misc cleanup

• version 214

  • wrote a new resizing dialog 'edit' class for simple resizing dialogs
  • added comprehensive frame size and position options for the new system to the gui options panel
  • moved review services frame to the new sizing system
  • windows that initialise maximised will correctly return to their last remembered size and position on a restore event
  • maximising a window by dragging it to the top edge of the screen should remember last position as the initial drag start position more reliably
  • positioning code is a little safer
  • fixed some missing recalculation of best/min size for wx.notebooks after page change
  • fixed missing recalculation of scrolledpanel's virtualsize after child wx.notebook's page change
  • fixed bad parentage for file import status frames
  • hid file import status button in manage import folders and subscriptions dialogs for non-Windows, as this is very broken, and the parentage fix wasn't enough
  • improved fuzzy padding on size calculations
  • refactored and cleaned and harmonised a bunch of the new window resizing code
  • suggested tags - favourites tag entry in options is now a live autocomplete dropdown
  • cleaned suggested tags - favourites layout in options
  • suggested tags listbox now sets its width to exactly fit its tags
  • improved workflow logic of removing/petitioning siblings and parents (shouldn't get stuck in loops as much now)
  • tag listboxes will update when tag siblings change
  • the 'auto-replace siblings' state on manage tags will no longer incorrectly apply to removal actions
  • import status caches now display errors in a more straightforward way
  • errors sent to import status caches are now also printed to the log
  • simplified how database exceptions are caught and displayed
  • database exceptions now preserve the original exception type
  • fixed db-side traceback line spacing in database exceptions
  • improved general database exception rendering
  • fixed imports for videos with negative start time
  • deleting videos from the trash that are currently rendering should be more reliable
  • crash.log now goes to the db dir, unless that isn't determined yet or is unwritable, in which case the traceback goes to console

• version 213

  • created a new tag listbox for the new suggested tags control
  • added a extremely basic and prototype suggested tags control for favourite tags to options->tags and manage tags dialog
  • off-screen position detection is now more lenient to account for window managers that position slightly off screen to cut off border
  • new dialog sizing code adds a little more fuzzy padding on all min size calcs, reducing superfluous scrollbars
  • hydrus's custom listbook now informs parent scrollable panels of need to recalc scrollbars on page change
  • the new flexible dialog and frame now catch these events and will expand their size if they need to and have the space
  • the client options dialog now works on the new sizing system--it will automatically resize when you change pages
  • some frame resize save event code is improved
  • the media viewer is no longer aggressive about claiming focus
  • stacked manage tags panel buttons to allow for thinner dialog
  • misc dialog cleanup
  • corrected some bad logic that was excluding transparent gifs from the new thumbnail handling
  • all gifs and pngs will have their thumbnails regenerated on both the client and server, taking advantage of the recent thumbnail improvements
  • fixed a bug if you click cancel when choosing an hta path to sync a tag service with on manage tags dialog
  • fixed a bug where noneditable services (local tags and local booru) could sometimes become editable on manage services dialog. the edit buttons should now disable correctly on init, and there's an additional check in the add code in case of future problems
  • any superfluous local files/booru services will be deleted on update

• version 212

  • wrote a neat flexible system for recording and restoring a variety of window size and position information
  • moved the manage tags dialog to the new size and position system
  • moved the main gui and the media viewer to the new size and position system
  • the manage tags window launched from a media viewer is now a non-modal frame, allowing interation with the underlying media viewer while floating on top of it
  • the new manage tags frame commits changes immediately!
  • review services and file import status will now stay on top of the main gui
  • fixed some bad advanced content update dialog launch code
  • the hydrus tag archives in hta syncs can now be anywhere, not just in the client_archives folder
  • whole bunch of hta sync cleanup
  • paletted images are now dequantized to RGB(A) before thumbnail generation, enhancing scale quality
  • 'LA' (greyscale+transparency) images will now render with correct transparency
  • png files will now always generate png thumbnails (previously it was just ones with transparency)
  • trivial options changes are less frequently saved
  • rule34@booru.org now parses namespaced tags
  • fixed a bug in thumbnail resize error recovery code
  • fixed a bug in file repository petitions
  • updated to Pillow 3.2.0
  • misc cleanup

• version 211

  • added options for disk cache init and maintenance to 'speed and memory' page of options
  • abstracted the manage tags dialog into a general purpose 'apply' dialog and a scrollable manage tags panel
  • the 'auto-replace entered siblings' checkbox now remembers how it was last set
  • uncertain sibling tag counts will now appear like 'tag (1,234-1,456)', showing the range of possible results they represent
  • fixed a bug where siblings would be collapsed inside the tag manager object if a specific tag search domain was set
  • altering the search text on an autocomplete from 'tag' to '-tag' or vice versa should graphically update more reliably, and an exact-matching predicate will be promoted to the top of the list correctly either way
  • calls to upnpc will now kill the process and raise an exception if it hangs for more than thirty seconds
  • misc improvements and cleanup

• version 210

  • manage tags dialog now launches much faster when thousands of thumbs are selected
  • manage tags dialog now has a prototype 'do this for all' checkbox to make mass selected-tag editing a bit quicker
  • polished the new disk cache turbo mode, added free memory limits and more options
  • a bit of disk cache population now runs on client boot, just to kick things off
  • a full disk cache population now runs before repo sync
  • the client now keeps about 200MB of itself in the disk cache during idle maintenance
  • the update files in client_updates are now stored compressed
  • the log files and 'already running' files are now stored in the db folder, meaning hydrus shouldn't have to write to anything but db dir and temp dir
  • 'already running' files now delete themselves on program close if they can
  • the log directory is no longer needed
  • if there is only one booru to select, it will be auto-selected
  • client failed exit is less likely to hang, more likely to log error
  • client won't bother to save lest session regularly or on exit if this is not the default startup session
  • hotfixed an auto-vacuum bug
  • added a catch for any future vacuum mess-ups to save the db from collapse
  • added other catches for other close/reinit db calls
  • misc cleanup

• version 209

  • the advanced content update dialog can now be !aunched from manage tags, and will only affect the files the dialog was launched for
  • gave the advanced content update dialog a bit of a makeover
  • system:ratio now supports 'taller than' and 'wider than' operators
  • merged client thumbnails into the dynamic file directory structure
  • the client files manager now manages thumbnail paths exclusively
  • thumbnail import responsibility moved to the client files manager
  • thumbnail resizing responsibility moved to the client files manager
  • orphan clearing and thumbnail regeneration maintenance tasks moved to client files manager
  • 'regenerate thumbnails' now has an 'only do missing' mode
  • thumbnail regen on resize is faster
  • copy file to clipboard responsibility moved to the client files manager
  • media file and thumbnail deletion responsibility moved to the client files manager
  • if a full size thumbnail is missing but regenerated ok, the client files manager will now only produce one, better error message
  • cleaned some bad error detection and reporting code
  • cleaned some regular 'thumbnail size has changed' code
  • fixed and cleaned a mix of file and thumbnail path code
  • client files rebalancing and recovery will now efficiently move-merge rather than copy-merge
  • fixed some bad portable->absolute path normalisation
  • improved missing external storage location error reporting
  • server thumbnails are similarly merged into server_files
  • to keep things simple, the client no longer keeps track of number of local vs. remote thumbnails for file repositories
  • perceptual hashes are now calculated from original file, not thumbnail, and only for jpegs and pngs (videos had too many black frames etc...)
  • the vacuum maintenance popup will now only appear if a vacuum can be completed in the time available
  • slideshows will now yield to any open popup menu, not just those launched from their own canvas
  • wrote a prototype turbo mode and put it on the debug menu
  • misc cleanup

• version 208

  • split the path tagging dialog's complex widgets into simple/advanced panels
  • added filename and directory 1, 2, and 3 easy tagging, with namespaces to path tagging dialog
  • fixed up some buggy 'tags just for selected files' behaviour
  • ipfs directories now support a note
  • the ipfs directory note can be edited from review services
  • added 'all namespaced tags' and 'all unnamespaced tags' to the advanced content update/service-wide operation dialog
  • added 'clear trash' button to trash panel on review services
  • separate timestamps for file repo upload and ipfs pin will be displayed on single-thumb right-click menu
  • the correct file domain timestamp will be used for newest/oldest sort
  • db integrity check now scans all attached database files
  • broke up mappings petitions in the server db into more efficient 'deleted' and 'petitioned' parts
  • cleared out some static refs in prep for external db location
  • simplified a bunch of file mirroring code and improved error handling of directory mirroring
  • fixed a rare thumbnail resolution calculation bug
  • misc smaller fixes and cleanup

• version 207

  • finished ipfs directory creation
  • ipfs review services pane now shows current directory shares
  • cleaned up some content event responsibility in ipfs service code
  • fixed a file download bug when using ipfs 0.4.2
  • wrote some more ipfs help, for directories
  • popup messages can now hold arbitrary copy-to-clipboard text content
  • fixed the new listbook changes for Linux, which was having trouble tracking the client_data inside the sorted listbox
  • fixed an issue with the custom filter management dialog possibly duping the 'default' entry on ok, making impossible to again launch
  • offloaded tag parent generation responsibility in the a/c write tag pipeline
  • added a checkbox to manage tags dialog that will control if siblings are auto-replaced on entry
  • all the clunky 'add preferred sibling' workflow is removed from the manage tags dialog
  • fixed incidence tag sort order not initialising properly in the options dialog (and transferring to lexi (namespace) every time)
  • fixed too-eager quit during shutdown db analyze
  • fixed the db restorebackup call, which wasn't closing the gui correctly
  • the manage services dialog will now automatically re-check 'allow zero' on numerical ratings with only one star
  • misc cleanup

• version 206

  • if the currently focused thumbnail is removed from view, the earliest non-selected thumbnail will be remembered as a 'ghost' focus if the user then presses the arrow keys to continue to navigate
  • if the user presses an arrow key to navigate the thumbs when there is no known focus thumb, the first will be assumed
  • the listbook now supports duplicate display names
  • services of the same type can now have the same name
  • cleaned some listbook code
  • inbox or custom filters will now render first-file animations correctly on initialisation
  • fixed the 'change' event on the new timedeltabutton, which was not updating the thread checker on dialog ok
  • subscriptions will now naturally terminate their gallery sync on a 404, rather than dumping out with an error
  • fixed some false positive 'paths are different' testing in default directory mirroring code (used in db backup, restore) that was slowing these operations down
  • if database files have the same file and modified timestamp, they won't be re-copied in a backup or restore
  • client external storage location recovery/rebalancing will now skip over files with the same size and modified date
  • if you backup the client and have external file storage locations, a popup will remind you to back those locations up manually
  • did a quick pass over some help stuff, and added a 'can the client manage files from their origial locations' bit to the faq
  • created a simple no-reward patreon, wrote some help about it, and added links in the usual places
  • empty 'read' a/c dropdowns will be more careful about idly refreshing their system preds, hopefully leading to less accidental gui hang on large jobs
  • main gui menus will be more careful about refreshing themselves when called, hopefully leading to less accidental gui hang on large jobs
  • text snippets are stored inside the client db in a better way
  • added flat service directory structure storage support to the client db
  • fixed a mapping petition rescind sql typo
  • fixed the v198->v199 server update code
  • misc cleanup

• version 205

  • fixed v201->v202 update code, which v204 retroactively broke
  • wrote a new maintenance routine to regenerate the ac cache if it ends up miscounting or not generating correctly
  • fixed specific ac_cache generation code, which was double-counting because of recent file-add optimisation
  • client db maintenance now vacuums attached databases according to their own vacuum clocks
  • client will more aggressively vacuum and analyze if the client has been idle for an hour
  • client vacuum now prints success statements, with duration, to the log
  • analyze now produces an auto-dismissing vacuum-like popup message
  • the 'is maintenance due' check on shutdown now accounts for the new vacuum logic and includes analyze needs as well
  • shutdown maintenance will use more of its allotted time if needed
  • tag archive sync now always filters to local files on initial sync
  • if you import a tag archive with sha256 hash, you are now given the option to filter to local files or import everything
  • the occasionally crashtastic opencv.imread call is now completely replaced with PIL, let's see if it's too slow
  • the new-but-now-redundant 'disable opencv for static images' option is removed as a result
  • fixed a redirection location parsing bug that was affecting sankaku subs with >1000 initial files
  • fixed some bad analyze timestamp tracking code
  • fixed advanced content update code, which still had some mapping service_ids floating around

• version 204

  • current, deleted, pending, and petitioned mappings are now stored on service-separated dynamic tables
  • any tag service deletion or reset should now only take a few seconds
  • fixed a physical file deletion bug that seems to have affected (some?) versions of linux and os x
  • reintroduced a revamped 'clear orphans' database->maintenance routine for the client, which will delete or move orphaned files and delete orphaned thumbnails. run this if you are on os x or linux
  • improved misc file deletion code
  • fixed some multiplatform path conversion, and generally improved how paths are stored and retrieved
  • the network application of a parent tag to all files with the child tag now happens serverside, and only on petitioner-approval of a tag parent relationship. it still occurs locally to the uploader, but is now wrapped up in the parent commit
  • fixed some numtags search logic
  • fixed an issue when mixing system:age with any tag-based predicate
  • media_results db calls no longer require a file domain
  • 'hide inbox and archive preds if either has no files' now defaults to off
  • added an option to disable OpenCV for gif rendering
  • the selection tags panel now initialises with the correct tag domain (it was previously always starting as 'all known tags', which is not always true for session pages)

• version 203

  • thumbnail resize now happens on the fly--feel free to change it as often as you like, it takes no time at all
  • added 'lexicographic (grouped by namespace)' tag sorting to the selection tags box
  • added an option to disable OpenCV for static images under the media options page
  • panning with the shortcut keys now pans by a twelfth of the media size or canvas size, whichever is smaller
  • cleared the analyze timestamp cache for both client and server, which will force a reanalyze of the new db files on the next db maintenance run
  • improved some misc analyze code--primary keys are analyzed again
  • added an explicit analyze command to the client's database maintenance menu
  • the server now analyzes attached database files
  • improved an unhelpful server thumbnail error message
  • added an invalid NULL check-and-skip to hash cross-referencing
  • fixed some invalid a/c write dropdown search domain initialisation
  • fixed some borked zoom calculation code that was sometimes lagging the media viewer and leading to 100% zoomed images being sent unneccessarily and then being nastily scaled down
  • fixed a file import error if a synced tag archive is missing
  • updated sqlite for windows
  • misc cleanup

• version 202

  • fixed a problem with the v198->v199 update step
  • added a bad error catch in the vacuum step in v200->v201
  • added some rollback exception handling for unusual situations
  • the autocomplete caches under the client_cache subdirectory are dynamically folded into a single client.caches.db file, and that file is folded into the main db journal like the other attached dbs
  • ac_cache file and tag processing lag and edge-case autocomplete miscounting should be removed as a result
  • cache generation is optimised for empty services
  • specific file_caches' add and delete file commands now modify their tables directly, saving a whole load of time that was previously done superfluously filtering
  • the specific ac_caches' add mappings command is a bit more logically efficient
  • increased cache size for each attached database
  • folded all 'attached' db update code into the main update_db routine
  • folded all 'attached' db init code into the main create_db routine
  • made the vacuum checking routine safer
  • reduced analyze log spam
  • analyze maintenance breaks its larger jobs up better
  • analyze mainenance now analyzes external databases
  • the db connection now refreshes every half hour, to regularly clear out journal files
  • the hydrus db assumes it generally has exclusive control over its db, so it no longer wraps its read-only requests in transactions
  • session load now happens off the main gui thread, and media generation is broken into 256-file chunks, which reduces startup db and gui lag for large sessions
  • the youtube video downloader now lists a webm option if youtube provides one (they seem to offer 360p webms on everything(?) now)
  • the timedeltacontrol (where you set the period for the thread checker and import folders and so on) is replaced by the more compact timedeltabutton, which will launch a dialog with the old control when clicked
  • the server will no longer revisit old updates and cull since-deleted content, as this was proving cpu-burdensome and not helpful to future plans for update dissemination, which will rely on static update files
  • misc cleanup of some server db stuff
  • content update processing will not spam menu updates throughout, but only notify once at the end, which should reduce idle gui hang due to db access choke
  • the autocomplete tag entry will not refresh system pred menu during idle, which should reduce idle gui hang due to db access choke
  • shutdown repo sync will now report the update summary text and will not spam the intervening statuses to the log
  • moved timestamp out of the serviceless media_result and into the locations_manager, to reflect the database's knowledge of each current service having a different timestamp
  • local and trash timestamps are generated from the locations_manager now, and non-local file service timestamps will soon follow (e.g. 'pinned to ipfs_service 3 days ago')
  • if the timestamp is unknown for the current service context, it will not be displayed
  • file repositories will now only sync thumbnails when their updates are completely synced (to stop 404s from since-deleted files)
  • fixed a ( 0, 0 ) resize event bug that was sometimes causing borked media containers on media viewer shutdown
  • syncing to a sha256 tag archive will still import the data as requested, but a popup note will explain that as everything will be imported, further syncing is pointless., and the sync will not be saved
  • fixed a bug in hta export if you click cancel on the hash_type choosing dialogs
  • misc cleanup

• version 201

  • exported hash and tag master tables to external database files for both client and server
  • the client's 'mappings' table is split into 'current_mappings' and 'pending_mappings', saving about 20% of space and speed for almost all mappings operations
  • removed superfluous rowid column from client mappings tables, saving about another 20% of space
  • removed superfluous rowid column from server mappings tables, saving about 20% of space
  • to save time, service mapping counts now only aggregate current mappings, not current and pending
  • exporting to tag archive now only exports current_mappings, not pending
  • 'all known files' queries will more cleverly discern their current/pending search domain
  • tag count queries will more cleverly discern their current/pending search domain
  • cleaned up a lot of mapping-related query code, and any ugly status leftovers from current/pending union
  • num_namespaces is no longer tracked for tag services
  • unified clientside service deletion code, fixing several orphan-generation issues
  • db update will remove all mappings orphans created by a recent tag service deletion
  • some db settings init is harmonised into one location
  • fixed up some external db settings init
  • db connection settings are now applied to all attached db files
  • db vacuum code is harmonised into one location
  • vacuum cleans up after itself better
  • db vacuum is now preceded by a hard drive free space check
  • db vacuum can now predict how long it will take
  • if a vacuum fails for an unexpected reason, the db will now recover better
  • the db recovers from failed backups better
  • the server db will vacuum all its databases on a backup, except any that it thinks will take total vacuum time to more than five minutes
  • server backup will report roughly how long it took, clientside
  • significantly sped up system-tag-only 'all known files' queries for tag services that have many mappings
  • significantly sped up system:numtags for =0 or >0 when tag services have many mappings
  • the status bar has some improved grammar, reporting overall common mime as appropriate
  • the status bar reports single-thumbnail info text when one file is selected
  • fixed an image cache key type-matching bug
  • data cache now tracks objects with dynamic memory usage
  • the static image rendering-cache relationship has been slightly dejanked
  • added a couple of checkboxes to the options->gui page to hide/show thumbnail title banner and page indicator
  • the manage tags censorship dialog now defaults to the 'all known tags' service on init
  • slideshows will not progress if the current media is either a static image that has yet to render or an unpaused animation that has yet to run through once
  • service content package status messages now summarise their final commit status better
  • this nice new line is printed to the log as a record of repo sync rows/s speed
  • the client will shut down faster if big jobs are running

• version 200

  • added 'censor tag' to tag right-click menu
  • cleaned tag censorship a bit and moved from the gui down to the db layer
  • fixed tag censorship for multiple services
  • sped up tag censorship a bit
  • improved some tag censorship logic for unusual namespace situations
  • tag censorship now applies to siblings and parents, even in their management dialogs!
  • fixed the missing fit zoom when scrolling through zooms and when flipping between 100% and fit zoom
  • cleaned some canvas init code, which was loading bad initial zoom values due to other recent changes
  • restored standard static image zoom mode for cv-supported images to high, inter_area quality
  • cleaned up some image/animation container init code
  • cleaned up some image cache image tracking code
  • fixed ipfs pin, which was conflicting with the new ac caches
  • extracted client deleted_mappings to their own db table
  • prepped mappings processing to work with status-separate mappings tables--it is also a lot cleaner now
  • because of the deleted_mappings extraction, a bunch of mappings membership tests are much simpler now
  • upped the db cache size to a semi-ludicrous 150MB, which nonetheless seems to improve processing speed
  • improved how hydrus databases manage their db file locations
  • improved how the client and server databases backup/restore
  • improved some non-local thumbnail fetching code
  • converted some old YAML stuff to JSON
  • improved big job polite pausing code
  • cleaned up some clientside imports, and set the client to only import its server code if it needs to boot a local service
  • when an import folder needs to rename a file it is about to move because of a conflict, it will now add autoincrementing numbers, starting at 0, and before the file extension
  • added some missing shortcut actions (pan_x and set_media_focus) to the shortcut input entry dialog
  • tags and siblings that match the current WRITE autocomplete tags entry will no longer have doubled parents!
  • improved how system:limit is applied to searches

• version 199

  • added 'all known files' autocomplete caches, so manage tags dialogs' autocompletes should now always appear quickly
  • the now caches will be created on startup, which may take a few minutes
  • deleted the old tag autocomplete cache
  • wrote a faster initialisation routine for all known files caches
  • fixed an ac_cache cleanup typo
  • the client's (usually gigantic) mappings table is temporarily extracted to an external db file, which should have numerous benefits and perhaps a few drawbacks for now
  • the update will take a few more minutes, and will need lots of disk space, to extract the mappings
  • the server has a very similar external mappings table, so it will also need a while to update, but future admin backup calls will be a lot quicker, due to a much smaller vacuum
  • greatly optimised a mass-membership calculation in tag processing
  • reduced some db check-spam timer that was adding up idle cpu time because of recent ac_cache additions
  • fixed predicate parent sorting
  • if swfrender fails to generate a swf thumbnail within 60 seconds, the client will now dump out (rather than hanging indefinitely)
  • fixed some string-selection dialog layout rules
  • optimised some 'all known files' search queries
  • subscriptions will now append new urls in oldest->newest order, roughly preserving that order in the internal url cache
  • the 8chan thread watcher now accepts .json API urls directly
  • the repository processing routine will handle server_busy errors more gracefully
  • cleaned and refactored most media viewer 'canvas' code, dejankifying a whole bunch of related stuff and generally improving the object hierarchy
  • cleaned up and unified a lot of logic and responsibility for idle checking and shutdown maintenance stop time for some big jobs
  • misc cleanup
  • misc dialog cleanup

• version 198

  • ipfs download now parses directory multihashes
  • ipfs directory trees are thrown up on a new checkbox tree dialog for selection/filtering
  • created a new download_urls (i.e. multiple urls in one popup) popup call, which the new ipfs directory downloader uses
  • improved a very inefficient line of sql in the mappings fetch stage of query result building that was also used in the manage tags dialog. on my dev machine, fetching 256 files' mappings dropped from 2.5s to 9ms!
  • regular gallery pages and 'page of images' pages now have a 'paste input' button to mass-add newline-separated queries from the clipboard
  • fixed a typo in new txt_tags import folder code
  • fixed bad autocomplete predicate sorting
  • fixed some sibling autocomplete search_text matching logic
  • some byte-based value/range presentation will now correctly convert to bytes, rather than regular decimal numbers
  • improved linesep splitting code across the program
  • fixed get/setlabel calls across the program to get/setlabeltext, so ampersand characters are handled properly
  • wrote an 'all known files' autocomplete cache db and prepped a bunch of code for it--it should be easy to finish next week
  • added a new regex practise link to the regex dialogs
  • os x app release should display better on retina displays
  • misc cleanup

• version 197

  • on client boot, autocomplete caches for specific file/tag service cross-references are now initialised and populated. progress is shown on the splash window
  • on client boot, surplus autocomplete caches are deleted
  • on service add, new autocomplete caches are created
  • on file add/delete, autocomplete caches are updated
  • on mappings pend/add/rescind pend/delete, autocomplete caches are updated
  • the new autocomplete caches are consulted for all non-'all known tags' queries
  • the old autocomplete cache no longer stores counts for specific file services, and the remaining associated maintenance calls are deleted
  • databases now start their own mainloops
  • databases now wait for their mainloops to finish prepping any large caches before they return to the controller
  • the client database waits for autocomplete caches to finish before it finishes its own mainloop
  • the padding around flash and the animation bar are included more accurately in some media zoom calculations, which should eliminate some general zoom jankiness and accidental 100% flash zoom coincidences that filled up the whole canvas
  • fixed some clientside server boot error spam when local server or booru had no port set
  • account refreshes that fail due to a network error will spam less to the log
  • fixed .txt unicode tag parsing from import folders, which was not decoding at the correct step
  • administrator immediate repository syncs now sync thumbnail downloads if needed
  • service thumbnail sync will no longer superfluously check the presence of thumbnails whose files are local
  • if a tag entered into the manage tags dialog has a sibling that already exists for all files, then a new 'ignore, as the sibling already exists' choice will appear
  • fixed an overcounting bug in 'selection tags' when importing and adding tags at the same time
  • fixed a typo in repository sync status text that was overcounting total number of updates by one
  • fixed youtube downloader, which broke with the new library on my new dev machine
  • the way that tags and predicates are filtered against a tag autocomplete text entry is now much faster
  • bumped up the default content update chunk threshold from 100 rows to 5,000, which seems to be speeding up processing significantly, with a cost to recovery latency--see how it works for you

• version 196

  • fixed the 8chan thread watcher for boards that host content on media.8ch.net
  • improved the thread watcher url check logic so it won't lag with the new fix
  • cleaned up the ac generation code a little
  • 'all known tags' ac counts are now summed from all the known tag services rather than calculated directly (a <= indicator for when these cases overlap will be forthcoming). this speeds up file add/delete, service reset, a/c fetch time, and general tag processing, and reduces the size of the db
  • ac generation code now deals with 'is the entry text an exact match or not?' better
  • ac generation code will now no longer produce non-exact-match siblings on an exact match search
  • ac generation code will no longer save half complete search text into the db as new tags
  • on update, the a/c cache and its helper table 'existing tags' will be cleaned of a lot of orphans, which may take a few minutes
  • fixed some bad unicode path parsing when importing files in some OSes, I think!
  • fixed some bad read autocomplete sibling substitution
  • fixed a bug where autocomplete predicate lists would not update if the new list was merely a reorder (which can happen in some unusual sibling cases)
  • fixed the tumblr parser for the subtly new API
  • import folders now support loading tags from neighbouring .txt files--check the dialog to set up which tag services you would like to import to
  • the ipfs file downloader now queries DAG object links, determines if the given multihash is a directory or other complicated object, and if so politely dumps out (handling of directory downloads is forthcoming)
  • some db code is cleaned up
  • prepared db code for some future subclasses
  • wrote most of the new ac cache db
  • misc cleanup
  • added some browser addon links to the ipfs help

• version 195

  • wrote up a v1.0 ipfs help page
  • added ipfs services to 'system:file service'
  • added a 'multihash prefix' option to ipfs services in 'services->manage services'--it will prefix the given text to multihashes copied to the clipboard (e.g. you could use 'http://127.0.0.1:8080/ipfs/')
  • fixed a bad repo sync processing bug that was incorrectly assuming packages were completely processed when they often were not
  • all repos will have their processing timestamps reset on update, forcing a (fast) reprocess of everything they have to cover the previously missed content
  • fixed 8chan OP image parsing, which was sometimes failing due to absent md5s in the json
  • widened the splash screen a little more so repo sync text can fit a bit better
  • a couple of splash status texts are shuffled around
  • updated to sqlite 3.11.0 for windows
  • if sqlite3 reports at least version 3.11.0, the db will stick to WAL for large transactions (lke vacuum and content processing), since these are fast now
  • 'namespace:*anything*' will now only appear if the a/c input has no 'subtag' component., e.g. 'character:'
  • import and export folders now print simple summaries to the log if they do some work
  • tidied up the 'pages' menu
  • the various hardcoded 'delete' key events are now also triggered by a backspace on os x
  • added a rough 'copy known urls' to the regular thumbnail menu. this is prototype, let me know how you would like this information managed and displayed in future
  • in prep for the new cache layer, the autocomplete dropdowns (and hence search domains of all contexts) no longer support 'all known files' cross referenced with 'all known tags'
  • misc cleanup

• version 194

  • ipfs pins and unpins can now be queued up like file repository pending and petitioned, through the regular thumbnail right-click menu, which also reports some/all ipfs pinned selection status
  • this ipfs action queue is similarly summarised and commited at the normal service 'pending' menu
  • ipfs's 'pinned', 'to pin', and 'to unpin' statuses are displayed on thumbnails with ipfs-specific icons
  • you can copy the focussed file's ipfs multihash or all the selected files' ipfs multihashes from the thumbnail menu's share->copy->ipfs multihash
  • added a .txt tag parser to the 'path tagging' import dialog--it will parse the same sort of txt files the export dialog produces
  • the client's new 'requests' network code is harmonised, generally improved, and now produces hydrus-compatible exceptions
  • updated help re the local server and boorus now defaulting to off
  • db can now remember service-specific filenames (e.g. ipfs multihashes)
  • cleaned up some overly complicated and confused thumbnail menu code
  • the pending menu now specifies what it is about to do more plainly

• version 193

  • the client's local server and local booru can be turned off from their respective management panels, and from now on, the client will initialise with them this way.
  • if the local server or the local booru are not running, their copy/share commands won't appear in right-click menus
  • the welcome dialog is now a simpler popup message
  • incidence sorted tag lists are now sub-sorted by a-z lexicographic
  • pasting many tags that have siblings to the manage tags dialog will ask you if you want to always preference the sibling, saving time
  • added a 'clear deleted file records' button to the local file service on the review services window
  • idle mode now cannot naturally engage within the first two minutes since client boot
  • the autocomplete search logic will not count namespace characters in the autocomplete character threshold, so typing 'character:a' will not typically trigger a (very laggy) full search
  • putting a '*' anywhere in an autocomplete search_text will force a full search, ignoring the a/c character threshold
  • moved some specific 'give gui time to catch up' pause code to the generalised pause/cancel code that a lot of stuff uses, so big jobs should generally be a bit more polite
  • split the daemon class into two--one for big jobs that remains polite, and another for small jobs that triggers regardless of what else is going on. this should increase responsivity for a number of scenarios
  • fixed some bad wal failure detection and hence no-wal file creation on some instances of db cursor reinit (usually after service modification). because of now many superfluous no-wal files, existing no-wal files will be deleted on db update
  • some external storage location errors are improved
  • some internal and external storage location init is improved.
  • if an error is detected in the external storage location manager, it will not attempt to rebalance again until the client is rebooted
  • improved some upnp error catching
  • cleaned up some misc shutdown thread-gui interaction error spam
  • did some prep work on a future rewrite of daemon jobs pipeline
  • split up some mixed file/data/404 'stuff was missing' exception code

• version 192

  • added a 'check on ok' button to the manage subscriptions dialog's subscription panel
  • check file integrity now prints missing paths to the log
  • fixed a typo that was breaking some repo sync download resumes
  • if an external storage location does not exist on client boot, a popup will say so
  • resetting a service is now much faster when the service is large
  • some repo processing stuff is a bit faster
  • improved some image rendering error handling
  • some thumbnail errors are recovered from in a better way
  • improved db update bit rot resilience
  • improved the vacuum disk space error popup with better explanation of the problem and solution
  • misc improvements
  • moved to win 10 dev machine, so several windows libraries are a bit newer
  • updated to opencv 3.1.0 on windows

• version 191

  • added ipfs service type
  • added simple ipfs service gui to manage and review services windows
  • added simple ipfs download and import to pages menu
  • split confused service data object into a bunch of more flexible and readable subclasses
  • raw url downloads are now powered by 'requests' rather than my hacked http engine, and work a hell of a lot better. more to follow
  • raw url downloads now pause and cancel correctly
  • optimised some a/c cache clearing in mapping update processing
  • optimised content update object -> db yield logistics
  • added no-daemons command line switch to disable all daemons
  • added no-wal command line switch to disable WAL journalling for the db
  • if the db fails to read a new WAL-journalled db file it has just created, it will delete the file and attempt to recreate the db, never trying WAL again
  • changed the way the db writes big repo syncs to disk so that the job can be abandoned much more quickly
  • improved local file not found error to state the directory or exact path that was the problem
  • fixed some socket-level error reporting (I think timeout was part of this)
  • the petition approval and denial client-to-server network commit is broken up into smaller steps to reduce server lag and the overall likelihood of timeouts
  • subscriptions will no longer add files to their 'successful files' button when the file's url seemed new but it actually was previously deleted
  • subscriptions will now identify themselves with an additional popup message when they encounter critical errors during sync
  • editing a clientside service will force a reset of the appropriate session cache, so if you change access key, the account changes will be reflected immediately
  • the import files dialog will no longer spam error popups on uninteresting or empty files--its 'parsing complete' statement is more verbose instead
  • the 1.5s gap between parsing complete and the buttons being enabled is also removed. this was supposed to be helpful, to give you a moment to check the files were as expected, but in the end it was just annoying
  • fixed a typo in some thumbnail error catching and regeneration
  • improved my network code syncing, so linux and os x releases are cleared out of many old files
  • misc code cleanup

• version 190

  • fixed some hashing recalculation in collections that meant they reported no files in many cases
  • fixed some hashing recalculation when 'remove files' is called
  • improved the way the client db stores file information and service->file mappings
  • idle processing jobs will now explicitly wake up as soon as the client naturally switches from not idle to idle
  • the minimum allowed value for the 'max cpu %' control in the maintenance and processing options panel is now 5%
  • the maintenance and processing panel is rewritten and laid out a little clearer
  • 'busy' is now 'system busy' on the status bar
  • force idle and force unbusy are now merged into a new 'force idle' that sticks until you explicitly turn it off
  • busy and idle states should now update immidiately after closing the manage options dialog
  • improved exit code event order to be less rude to the OS
  • improved exit code emergency event handling
  • fixed a typo that was not appropriately skipping the 'do you want to run shutdown jobs?' dialog
  • file storage folder rebalancing will now occur on shutdown maintenance
  • the client now closes down more reliably if the db fails to boot
  • the client now closes down more reliably if the gui fails to boot
  • if a client vacuum fails, it'll now also raise the exact error sqlite gave
  • fixed ctrl+a on the autocomplete dropdown (I think the Insert/IME support change broke it)
  • the ways the 'read' tag autocomplete control talks to other gui elements is improved
  • the tag autocompletes will now refresh their results lists on more internal variable changes
  • the query page management controller manages fewer redundant variables
  • updated sqlite for windows
  • the client and server dbs will attempt to change to TRUNCATE journal mode if WAL causes a read disk i/o error
  • misc code cleanup

• version 189

  • split the big analyze db calls into individual table/index calls and moved them from update code to the normal maintenance routines
  • on vacuum, both the client and server dbs will now bump their page size up to 4096 if they are running on windows (server vacuum is triggered by running a backup)
  • vacuum should be a slightly faster operation for both the client and server
  • boosted the db cache significantly--we'll see if it makes much difference for things.
  • the way the selection tags control updates its taglist on increases to its media cache is massively sped up. An update on a 5,000-thumbnail-strong page now typically works in 3ms instead of ~250ms. large import pages should stream new results much more quickly now
  • sped up some slow hash calculation code that was lagging a variety of large operations
  • some hash caching responsibility is moved about to make it available for the add_media_results comparison, which now typically works in sub-millisecond time (was about 16ms before)
  • some sorted media list index recalculation now works faster
  • some internal media object hashing is now cached, so sorted list index regeneration is a bit faster
  • some medialist file counting is now superfast
  • wrote a new pauser object to break big jobs up more conveniently and reduce gui choking
  • the repo processing db call now uses this pauser
  • some copy and mirror directory functions now use this pauser
  • backup and restore code for the client now skips re-copying files if they share the same last modified date and file size
  • backup code for the server now skips re-copying files if they share the same last modified date and file size
  • http cannotsendrequest and badstatusline errors will now provoke two reattempts before being raised
  • socket error 10013 (no access permission, usually due to a firewall) is caught and a nicer error produced
  • socket error 10054 (remote host reset connection) is caught, and the connection is reattempted twice before being raised
  • the old giphy API is gone, so I have removed giphy
  • forced shutdown due to system exit/logoff is handled better
  • pubsub-related shutdown exceptions are now caught and silenced
  • an unusual shutdown exception is now caught
  • fixed a copy subtag menu typo
  • cleaned some misc hydrus path code
  • tags that begin with a colon (like ':)' ) will now render correctly in the media canvas background
  • some misc code cleanup
  • dropped flvlib since ffmpeg parses flv metadata better anyway

• version 188

  • if you have custom filters set up, they will now be listed in the normal thumbnail right-click menu, which will expand for them
  • 'refresh account' button's event no longer waits for service response on the gui thread
  • 'refresh account' button on review services will now re-enable if the call fails
  • added 'hide inbox and archive predicates' checkbox to options->default file system predicates. not sure if this is the best place for it!
  • system:hash can now query md5, sha1, and sha256
  • existing pages with system:hash will break on update. since this is such a rare predicate, I assume this is not a problem!
  • improved db hash cross-reference flexibility
  • any pausable popups will now explicitly change their text to 'paused' when they are paused
  • upload pending popup now has a title, so it should pause a little more sanely
  • regenerate thumbnails popup now has a title, so it should pause a little more sanely
  • cleaned up some job pause status logic
  • upload pending now works on a stream, so gigantic uploads will cause fewer hitches and overall problems
  • upload pending code is cleaned and simplified in several ways, with better error handling and progress reporting
  • synctotagarchive is now split into a thousand tiny separate jobs and runs on a separate non-db thread. it will no longer lock the db the whole time
  • synctotagarchive popup should be a lot more responsive and will specifically give the gui some time to catch up after every chunk of data processed
  • synctotagarchive will update pending tags count (and gui stuff will show the pending tags) as it goes
  • synctotagarchive is now a pausable and cancellable operation!
  • large repository sync jobs should be a little less laggy on final commit
  • system:duration now displays its value as a properly formatted milliseconds time delta, rather than just a big int
  • profiles now flush to the log immediately
  • some cpu-heavy daemon recheck periods have been extended to reduce potential hangs on slower computers
  • some repository sync messaging wording and logic is improved (it should report what it is doing more reliably when you are cpu-busy)
  • reordered instances of 'waiting politely', so download text/gauge updates should occur before the wait does
  • some misc 'waiting politely' logic is improved
  • pixiv ugoira is detected and a better error (that will be ignored by the new sub error throttling code, for instance) is produced
  • 'uninteresting mime' errors have nicer import status note text
  • fixed the 'thumbnail grid background' custom colour option

• version 187

  • file->restart now works for linux
  • the 'call to' thread pool now has a delayed start, which fixes a shutdown hang in linux (and perhaps the other platforms, in one way or another), when the user hits 'forget it' on the 'already running' boot dialog
  • removed some console error junk on abandoned client boot
  • if a requested full-size thumbnail is missing, the client will now attempt to regenerate it
  • if a requested full-size thumbnail does not render (for instance, during a request to generate a resized thumbnail), the client will now attempt to regenerate it
  • some thumbnail fetching and generation logic is slightly improved
  • you can now drag animated media around when the inital drag click begins over the animation canvas
  • ditched some ultimately unhelpful mouse warping when the cursor was near the edge of the media viewer when a drag began
  • possibly fixed a strange tags_fts4 mirroring problem that has hit some users for a reason I do not understand
  • import tag options panels will no longer order their tag services randomly
  • fixed a bug where import folders would not save changes to their explicit tags
  • fixed a bug where under some circumstances, the gallery importer was attempting to display previously deleted files on the screen, causing errors.
  • subscriptions will now pause a few seconds after a file import error, which should solve a problem with client hanging when lots and lots of subscription file errors occur one after another
  • if subscriptions encounter 5 file import errors, they will pause their sync and try continuing later
  • pixiv manga error handling is improved
  • bad mime error handling is improved for all importers
  • fixed an important bug in the import status cache where error texts were not being stored as unicode, so error texts with non-unicode-compatible characters were causing exceptions when the session was being saved to the db or the import status frame was being loaded
  • existing sessions' import status caches should automatically convert their 'notes' to unicode on update
  • removed booru db fetch spam when booru download page has no work to do
  • fixed a typo in regenerate thumbnails popup message
  • fixed some bad event handling for the gallery page's file limit control
  • updated some regex defaults to have better multiplatform support and handle a wider array of valid characters

• version 186

  • Windows release is now 64-bit!
  • some libraries got updated for windows, to varying minor effects
  • BeautifulSoup updated to latest version for everything
  • fixed a warning from newer versions of BeautifulSoup by explicitly using lxml
  • updated the running_from_source help file to what is currently needed
  • ffmpeg.exe is also 64-bit
  • wrote a "reducing program lag" page for the help
  • added program restart to file menu, doesn't work for linux yet
  • improved restart code (it was used in restore database, and for some users would not close the parent process down properly)
  • in the media viewer, page up/down, arrow keys and mouse scroll no longer unhide the mouse
  • in the media viewer under Windows, dragging no longer unhides the mouse
  • the implicit delay in the downloader-importers is reduced, so redundant files should always stream in a super fast now
  • export files dialog will now auto-create missing destination directories
  • export files dialog will now work off the gui thread (to stop gui hangups on big jobs) and will report its export status through the export button, which will be disabled until the job is finished

  • changed the new tab/shift+tab shortcuts because they break panel control navigation (I want to make these editable soon)

  • 'fetch autocomplete results now' shortcut is now Ctrl+Space (Raw Control for OS X, not command)
  • 'IME mode toggle' is now Insert
  • cleaned up how control is handled cross-platform--in general, 'control' is always control for windows and linux, and 'control' is command for os x. this may be incorrect.
  • json dump errors now report more information about the specific object and its serialisable contents that are causing the problem
  • clipped sibling/parent reason entry dialog message when lots of pairs are added
  • fixed a bit of bad logic in sibling/parent dialog->content-update workflow
  • fixed the thumbnail canvas not refreshing on a canvas shrink caused by expansion of the management panel that doesn't change the thumb column count
  • fixed emergency boot error reporting for when HydrusData won't load
  • added some unicode conversion unit tests

• version 185

  • right clicking on a tag/predicate in the 'selection tags' box or the 'search' list of active predicates provides new intelligent menu options to discard/require/permit/exclude the selected tags from the current search. try it out!
  • this system also works for namespace and wildcard and system predicates
  • 'predicate a' and its inverse '-predicate a' are no longer considered the same!
  • namespace and wildcard predicates will now remove their inverts if they are added to the list of active predicates
  • the way inverse predicates are generated and compared is generalised and improved
  • fixed 'open a new search page for tag x', which might have been broken?
  • quick-entering a tag in the 'read' tag autocomplete entry will now always replace with a sibling if one exists
  • quick-entering a tag in the 'write' tag autocomplete entry will now not replace with a sibling--you will get the option of what you want to do
  • when this happens, the preferred sibling is labelled on the 'what do you want to do?' popup in the manage tags dialog
  • also when this happens, the affected file count for add/pend the preferred sibling should now be correct
  • fixed client support for tags that begin with ':', such as many emoticons. these should now be parsed correctly from websites and keyboard input, keep their leading colon through network and db conversions, and display correctly (for the most part!), despite a little bit of magic to make them work
  • created a new TimeDelta control to handle finer tuning of optional periods
  • improved timedelta->pretty string code to support >60s times
  • manage subscriptions dialog now supports day and hour period, with min period of four hours
  • manage export folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately
  • manage import folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately
  • the thread watcher now supports hour/minute/seconds check period
  • folded 'search' and 'download' menus into 'view' and renamed it 'pages'
  • 'sessions' moved from 'file' to the new 'pages'
  • moved 'admin' menu to 'services->administrate services'
  • added 'forced system:limit' to options->speed and memory panel
  • if several system:limits exist in a search, the minimum will be used
  • added a checkbox to options->downloading to replace the traffic light waiting politely indicator with text
  • subscriptions with failed files will say 'x urls in cache, y failed' on their dialog panel
  • at the gui level, media that superfluously receives a 'pend tag' instruction for a tag that is already current will discard the instruction (this improves the accuracy of the pending tag count during and after the manage tag dialog)
  • and the same thing for 'petition tag' when the tag isn't already current
  • any accidentally added 'namespace:'-type tags will be deleted on update
  • fixed a bad merge in the manage tag siblings dialog's autocomplete dropdown lists
  • the thread watcher can now deal with urls with a #54951495 post anchor at the end
  • refactored some controller manager code
  • removed some useless old code
  • cleaned some misc code
  • improved some bad old orphan deletion code
  • deleted some old code
  • finished off some old media result streaming code
  • refactored a bunch of search stuff from clientdata to clientsearch
  • rewrote some subs gui/help text to be a bit clearer, and added a help link to the dialog

• version 184

  • added external client_files storage!
  • you can add external client_files folders in options->file storage locations, further giving them weight
  • a new daemon will incrementally rebalance your files (and recover orphaned subfolders!) over your different storage locations
  • you can also force a full rebalance from the new database->maintenance->rebalance file storage
  • simplified the maintenance and processing panel
  • the maintenance and processing panel controls now appropriately support being set to 'none'
  • the 'run jobs on idle?' question is now explicit on the maintenance and processing panel
  • deselecting 'run jobs on idle/shutdown' will now disable subordinate controls
  • hitting tab on the autocomplete control now triggers an immediate 'fetch results' call
  • added a checkbox to options->speed and memory to completely disable automatic autocomplete results fetching (i.e. if you want to manually control a/c result-fetching only with this new tab shortcut)
  • the new less-laggy autocomplete results fetch won't trigger if the latest query is shorter than the cached query (i.e. you won't get lag when hitting backspace a bunch of times on autocomplete)
  • hitting shift+tab on the autocomplete control now disables all other key event capture, letting you enter IME without your up/down/enter presses controlling the dropdown list of results
  • 'waiting politely' time is now indicated with a small 'traffic light'-type circle control on all downloader pages
  • fixed some occasional 'I'll go sit in the top-left corner of the screen and not fix my position' hover window behaviour in Linux
  • fixed sibling predicate collapse for the 'read' autocomplete dropdown for database results
  • refactored how predicates are collapsed and sorted to be a bit more sensible
  • fixed maintenance of local booru data use even if local booru does not receive any requests
  • improved some service/content update error handling in repository sync
  • the system:rating value-entry dialog no longer lists its rating services in random order
  • 'open externally' will no longer show for non-local thumbnails' right-click menus
  • fixed audio/pdf thumbnail display for local booru, although they will be full size for now
  • harmonised some hex-prefix folder generation code
  • improved some file and directory copying code
  • improved upload pending popup message cleanup
  • added help_dir to hydrusconstants
  • miscellaneous refactoring
  • misc cleanup
  • extracted server-specific services and server resources from the common import path to the server import path
  • a bit more misc server refactoring
  • updated a couple of bits of help

• version 183

  • added swf thumbnail support--it works ok for most swfs!
  • thumbs for existing swf files will generate on db update for both client and server
  • the server will also generate thumbnails for video on update, which it seems it was not doing before
  • rewrote some hash comparisons in thumbnail downloading and thumbnail counting code to be a lot faster and memory efficient
  • fixed thumbnail count invalidation after service sync
  • in certain cases, fetching autocomplete tag results from the db will be slightly less laggy
  • if the autocomplete dropdown has cached or otherwise quickly fetchable results, it will refilter for current input as you type, ignoring the normal character delay (i.e. typing through autocomplete is far less laggy now)
  • fixed a couple of autocomplete tag-parsing bugs for tags with more than one colon character
  • fixed some key event selection bugs in the autocomplete taglist when there are no results in the list
  • if there are no results in the autocomplete taglist, the typically caret-moving key events up/down/home/end will move the caret (rather than being uselessly passed on to the empty taglist)
  • all profiling now goes through a single location
  • profiling now also prints extensive information on explicit code callers
  • fixed db profile mode printing display for the new logger
  • added new pubsub_profile_mode (AKA log-killer)
  • all menu popup display and explicit memory cleanup is done through a single location
  • hover windows will now not show whenever a menu is open
  • hover windows will now not hide whenever a menu is open (useful when trying to go 'copy tag' off the hover taglist's menu!)
  • a keyerror bug when force_idle_mode was hit right at program start is fixed
  • unexpected serverside client disconnections are now caught and further request processing is cancelled
  • middle clicking on an 'empty' system predicate in the a/c dropdown now will throw up the dialog to enter a value and then pass that filled in system pred onto the new search page
  • duplicate parents (which can occur with overlapping grandparents) are now collapsed
  • the upload pending popup message cancels properly after 'server busy' events
  • whenever the client's services are iterated at the view-level, it will now be in random order, to stop sync bottlenecks choking other services (and anything else like this that might have been occuring)
  • fixed a bug with double-clicking a tag selection on the the tag censorship taglist
  • fixed the too-fast frame timings on some videos opencv had a problem parsing
  • loading many video types will now be just a little faster
  • remote files will no longer present the right-click menu option to share->copy->file
  • hitting a shortcut for manage ratings from the media viewer canvas will no longer work if you have no ratings services
  • the base scripts (client.pyw, server.py and test.py) now have a shebang line to facilitate non-Windows use
  • laid groundwork for external client_files locations
  • plenty of misc cleaning and refactoring

• version 182

  • all printing to the log should now be unicode safe
  • some other, miscellaneous file-write locations should now be unicode safe
  • cleaned up some now-redundant unicode-bytestring conversion error handling
  • the animation scanbar will now draw a little darker while the animation is paused
  • the client now uses the same new log object as the server, so all logged data will be copied to the terminal, and all logged statements will be timestamped
  • the client and server's boot and exit statements are more harmonised
  • server backup is more log-verbose
  • server backup now makes a simple copy to 'server_backup' folder--no more _backup nonsense
  • all serverside requests will now print a line in the form 'PORT METHOD PATH HTTP_CODE in TIME TOOK' to the log
  • page up and down now work (again?) for the thumbnail view. adding shift also works for selecting a page at a time
  • improved some animation painting logic
  • improved some rating control painting logic
  • improved splash screen painting logic
  • improved all other, misc painting logic--many things should flicker less or render just a little quicker
  • ratings hover window will no longer re-layout (causing flicker) on ratings set
  • improved some media canvas painting logic to de-jaggify some zooming
  • in the manage options dialog, tag-related gui options are moved from the gui panel to the new tags panel
  • added 'apply all parents to all services' option to the tags panel
  • the delete key now removes active search predicates and tags in the manage tags dialog
  • fixed webm link parsing for rule34hentai.net
  • if you attempt to petition multiple tags, you will now be presented with a dialog asking you if you want to use the same reason for all the petitions
  • edit import folder dialog now has scrollbars and will resize itself based on your monitor size
  • if the static ffmpeg executable is absent from the bin folder, the client will now attempt to just call 'ffmpeg' in the normal system path
  • fixed some 'None media' data calculation bugs when media viewer closed during very fast slideshow
  • fixed a 'None media' mouse position bug when media viewer closed during very fast slideshow
  • all wx timers will explicitly stop on exceptions (which should reduce some types of error spam)
  • refactored client hydrus network session manager to a better position
  • wrote a better wma/wmv determining test using ffmpeg
  • refactored cv2 (OpenCV) out of the server's import tree
  • fixed some server boot crash error handling
  • cleaned some autocomplete matches-compiling code
  • deletepath and recyclepath will no longer throw an error if the path does not exist
  • server_messages folder is no longer referenced in the code--if you have it, feel free to delete it
  • memory cleans up a little faster after gui page deletion

• version 181

  • fixed a potential bug in the server's db, very important you update this week if running on Windows
  • improved some thumbnail page calculation logic
  • improved some thumbnail page drawing logic
  • fixed broken vertical resize thumbnail grid issue
  • fixed some broken 'thumbnails have changed dimensions' event layout recalculation
  • fixed an idiotic typo bug that was making cached thumbnail page bmps taller or shorter than needed
  • thumbnail fading and other general thumbnail redraws should be a decent bit faster now
  • thumbnails have handed off all bmp storage responsibility to the thumbnail cache, which should mean greatly reduced memory use when browsing very large results
  • thumbnails will now only quick-draw if they have been seen before on the same page, which should reduced some page-refresh waterfall jankiness
  • thumbnail media panel double buffering seems to be working better
  • improved some general thumbnail drawing code
  • the media viewer background taglist will now show petitioned tags
  • the media viewer hover taglist should now look exactly like the background taglist (sorted lexicographically, no more counts, and now shows petitioned)
  • the 'selection tags' taglist will now show petitioned tags for 'all known tags' queries
  • the 'selection tags' taglist and the media viewer background and hover taglists will now count collapsed sibling tags correctly
  • all taglists now support ctrl+a to select all
  • improved some generic taglist value-setting code
  • parents should now only expand for their specific tag services
  • siblings, parents, and the pending count should now be correctly recalculated upon repository update processing
  • added 'copy sources' to the import status window's right-click menu so you can copy bad urls or whatever to the clipboard
  • added 'skip' to the import status window's right-click menu so you can skip urls you don't want
  • added default booru for rule34hentai.net
  • under certain circumstances, services with duplicate names could be created--any services like this will be renamed on update, and the loophole where this could happen is now closed
  • improved a locale number formatting call to ask for unicode
  • improved newgrounds parsing--mp4/wmv files should now work
  • the server logger has been formalised into a proper class
  • fixed some bad static image initialisation in the canvas code that was causing error spam
  • fixed default import tag options for hentai foundry and pixiv
  • copy->(bmp|path|local url) will now only appear as an option if the currently focussed media is local
  • fixed some gallery identifier print code
  • frames and dialogs will now clip their initial size to the current monitor's resolution--better scrollbar support for cramped windows will follow
  • updated ffmpeg static builds
  • updated sqlite command line exes
  • put off sqlite windows dll update because of critical server db bug
  • misc code cleanup and redaction

• version 180

  • middle click on any taglist now does 'open a new search page for selection'
  • moved mouse idle checking code to the wx thread, which seems to have fixed a serious X11 crash
  • memory gunk should now be painted over when transparent animations are resized on linux
  • fixed some page up/down index calculation on the autocomplete dropdown
  • system predicates will remove themselves from the active predicate list silently again
  • system predicates will work again in queries that include tags!
  • greatly improved my string/unicode conversion code
  • dropped many inefficient unicode conversions from db code
  • improved the sql-grammar of some specifically efficient db code
  • removed incorrect unicode conversions from url/network request code
  • removed unicode conversion code from network response code
  • all directory iteration should now occur in unicode
  • all inputted paths should now be converted to unicode
  • replaced every single ugly raw path concatenation with a better method
  • improved some bitmap placeholder initialisation that was causing Linux terminal spam
  • added an option to always hide the preview window to options->gui
  • the preview window will not accept setmedia calls if it is hidden
  • if the preview window is actively hidden due to an unsplitting event, it will null its current media
  • fixed a shutdown calculation that was under-remembering the height of the preview window by four pixels every time
  • improved thumbnail waterfall code--it should be a little faster. less subject to lag when taxed, and waterfall with more genuine randomness
  • improved some thumbnail page drawing and clean/dirty conversion code
  • when pages are dirtied, any to-be-waterfalled thumbnails are cancelled, making superfast scrolling less crashtastic
  • new dirty thumbnail pages are now generated as needed, rather than precached
  • subscriptions will now correctly do a polite wait after fetching tags for redundant files
  • splash screen will now explicitly raise itself to the front on init
  • the export tag txts function of the export dialog now exports the tag txt to [filename].jpg.txt, rather than [filename].txt
  • the server now stores its updates in service specific subdirectories, like the client
  • some importers will better report when they are waiting politely
  • OS X (and anything else with OpenCV 3.0.0) now uses OpenCV to render gifs by default, falling back to PIL on error (like only Windows used to)
  • 'open externally' from thumbnails now defocusses the preview window
  • refactored away a bunch of bad global controller references
  • added some shutdownexceptions
  • silenced shutdownexception terminal spam
  • reworded the 'maintenace and processing' options panel a little to explain things better
  • improved how the base_dir is figured out on program boot
  • improved serverside 'should I clean dirty updates?' check code
  • deleted some old code
  • split path-specific functions off from HydrusFileHandling to HydrusPaths and de-looped my import tree appropriately
  • general refactoring

• version 179

  • all tag listboxes support multiple selection
  • all tag listboxes manipulate their data as sets of tags rather than individual tags
  • all tag listboxes report to their callbacks as sets of tags
  • many dialogs and other windows that use tag listboxes now deal with tags as sets
  • double clicking the media viewer's tag hover window now launches the manage tags dialog
  • the media viewer's tag hover window now includes (+1) counts like the manage tags dialog--harmonising the underlying canvas tag list will follow
  • lots of tag/sibling/parent code has been iterated over
  • ditched some misc redundant code and needlessly tightly coupled object relationships
  • some bizarro tag sibling dialog code has been rewritten
  • the autocomplete dropdown for 'writing' will now not expand search results to include parents when the receiving control isn't interested in them or it is otherwise not appropriate
  • when the autocomplete dropdown does have parents, they will be selected in a group with their child
  • the autocomplete dropdown for writing now broadcasts 'I'm done with tagging, close the dialog pls' in a better way
  • 'activating' the sibling or parent dialogs' listcontrols (usually by hitting enter) now processes all rows that are selected, not just the first
  • manage siblings and parents dialogs can now take multiple initialising tags from the taglist right-click menu
  • removed some redundant listbox code
  • cleaned up and improved some tag list event processing
  • cleaned up some taglist right-click menu code
  • added system-wide mouse idle test to idle calculation. you can set this in files->options->maintenance and processing, and it defaults to ten minutes
  • import folders now support tag import options' explicit tags for multiple tags to any tag service
  • existing import folders will be updated to the new version, and if a local tag exists, it will be intserted into the new import tag options
  • fixed a hentai foundry page parsing bug
  • the deviant art downloader can now download >1024 pixel width versions of images via the download button (and a bit of cookie magic)
  • gallery page queries that 404 (like for a non-existent username) will now report 'Gallery 404' rather than spamming the gallery's whole custom 404 html page to the status box
  • fixed a bad layout flag that meant some namespace checkboxes in tag import options could remain hidden during first panel expand until first mouseover
  • the import status frame now initialises its status text properly
  • fixed last week's num_tags optimisations, which accidentally broke num_tags < x for x > 1
  • the client should work on both OpenCV 2.4.x and 3.0.0 (thankfully, the only difference for our purposes turned out to be some static variable renaming)
  • the Windows and OS X releases now come with OpenCV 3.0.0
  • increased the max period of import and export folders to 30 days
  • the launchfile/directory thread is now a daemon, so the (some flavours of Linux) client can shutdown even if an externally launched file/dir remains open
  • misc cleanup

• version 178

  • import tag options now supports 'explicit tags', which will be added to all files imported
  • the default import tag options system supports explicit tags, and will propagate them to new import tag options objects
  • the gelbooru 503 Forbidden problem is fixed
  • all gallery downloaders now send the page_url as http Referer header when fetching their image_urls
  • pixiv parsing now grabs the correct tags for the image! previously, it was fetching the user's most common tags
  • for now pixiv manga (which is not yet supported), will be skipped over in the gallery parsing step
  • fixed a thumbnail drag-and-drop calculation that meant the DnD event would not start if the initial mouse movement went in an up-left direction
  • bad animated gif frame timing detection is improved
  • the fallback animated gif frame timing is scaled back to a more reasonable 12fps
  • fixed a critical bug in the manage ratings dialog that meant any NULL or MIXED numerical rating controls were mistakenly creating 'set rating to 0.0' content updates
  • fixed an obselete call with the new Pillow library that broke video thumbnail generation
  • added an explicit file import test to my test suite (previously, this was a manual test, and I did not notice the failed video files last week)
  • fixed a critical os x crash after navigating media with the manage tags dialog launched from the media viewer
  • improved speed and memory use of num_tags queries a little bit
  • reordered the predicate filters in the main query code so that num_tags happens right at the end, where it will run fastest
  • removed an unnecessary filter from the main query code
  • removed an accidental doubling of the simple predicate filters from the main query code where the file service domain is specified
  • improved an important db index and made sure the db uses it everywhere it can. it should speed up most non-'all known tags' queries
  • harmonised how all the new serialisable JSON objects are sent to and read from the db to a handful of simple general functions
  • externally opening a file is offloaded to a temporary thread to avoid gui lockups on OS flavours that have delayed 'open this file pls' calls
  • increased the commonly used read/write 64k block size to 256k, let's see if it reduces some file fragging
  • clientside hydrus bandwidth reporting is a bit faster and simpler
  • general code cleanup
  • general refactoring
  • a long-term refactoring job that aimed to better decouple the client, server, and hydrus parts is now completely done
  • delineated the layout of the gallery and page of images management panels a little
  • updated several links in the help and the program to https
  • the petition content checkbox list is now significantly taller
  • cleaned up the static folder a little bit
  • silenced an unhelpful webm last frame error

• version 177

  • the old YAML "default advanced tag options" is converted entirely to the new JSON "default import tag options"
  • created a new JSON object to store new options objects
  • updated the gui to handle new default import tag options objects
  • existing default advanced tag options data will be converted to the new system on update
  • the petition management panel now presents similar petitions from the same user together, with the relevant data rows in a checkbox list, so janitors can approve/deny many similar petitions all at once
  • the petetion management page no longer auto-loads file results for a petition--you can now see specific results for each sub-petition by double-clicking on them
  • the server-to-client petition object is converted from yaml to json
  • the new petition object now stores all the different 'content' rows for the same service/account/reason/action
  • small content rows are now stored and transmitted in their own object
  • the network now supports POST JSON requests
  • account modifications now work through JSON
  • the account identifier object is converted from yaml to json
  • in the path tagging dialog, regexes that do not compile will now say so, reporting their short regex error statement
  • fixed a bug that could sometimes occur when cancelling an ongoing gallery query
  • cancelling an ongoing gallery query is now reflected in the gui much faster
  • repositories will not burn CPU cleaning their updates if there are still pending file/mapping petitions to process
  • repositories will clean updates in smaller bursts to reduce the contiguous server_busy time
  • fixed drag and drop thumbnail ordering, which actually wasn't working but was accidentally randomly sorted when I tested it!
  • a certain kind of error during vacuum that can mean two different things is now caught, and automatic vacuums then suspended while the user figures out what is going on
  • revised some v176 update code that wasn't parsing some downloader pages correctly--now, if a downloader page cannot be updated, it will be discarded
  • updated to Pillow 3.0.0 on all platforms and fixed some obselete image calls
  • general code cleanup
  • general code refactoring

• version 176

  • the gallery downloader is updated to the new system
  • the gallery downloader code is generally a bit better all around
  • the gallery downloader will now remember its urls and file and tag options and paused status and so on throughout a reboot
  • the gallery downloader supports 'get tags even if file already in db' option
  • the gallery downloader management panel is updated for the new object. like the other panel updates, it is a little more minimalist
  • the gallery downloader management panel now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever
  • existing gallery downloader pages in gui sessions will semi-convert on the db update
  • subscriptions are updated to the new system
  • the subscription code is generally a bit better all around
  • subscriptions will no longer dump out on a failed file!
  • subscriptions can now recognise when files are uploaded during a page-walk (which shuffles everything up by one, of course), and recover without assuming it has synced
  • subscriptions can now be set an initial file limit and a periodic file limit separately
  • the manage subscriptions dialog is updated for the new object
  • the manage subscriptions dialog now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever
  • existing subscriptions will convert on the db update
  • subscription popup now shows per-file download progress
  • boorus that use number of thumbs to advance their gallery page indices can now resume
  • the thread watcher now obeys 'exclude deleted' status more reliably
  • the 'page of images' downloader now obeys 'exclude deleted' status more reliably
  • subscriptions now attempt to recover their sync status when a new file is uploaded during sync
  • fixed a bug where the thread watcher was not applying the filename namespace to new files, only redundant ones
  • the old advanced_import_options dictionaries are all updated to the new import_file_options object
  • improved subscription-dialog-was-opened-while-subs-were-running detection and reaction
  • the way galleries are identified behind the scenes is now much simpler and more flexible
  • for now, hentai foundry downloads and subs will no longer support the specific hf filter--they will be unfiltered and sorted newest first, so they work like all the other galleries. I may reintroduce 'sort by popularity' if there is demand for it
  • a bit of importer status update refactoring and cleanup
  • fixed some db vacuum maintenance timing that was interfering with gui reporting, so the 'currently vacuuming' popup never appeared until the vacuum was complete
  • fixed a possible bug for import folders that have a local tag set
  • splash screen is no longer stay_on_top--if there is demand, I will add an option
  • fixed a bug in the page deleted detection code
  • added page hidden detection code, replaced some ambiguous pause code with it
  • improved pause/play code throughout the program
  • corrected some bad x/y status texts
  • for simplicity, the import_file db command now always returns ( status, hash )
  • for simplicity, the import_file db command no longer takes service_keys_to_tags
  • plenty of general code cleanup

• version 175

  • the 'regex shortcuts' button in the path tagging dialog now permits the managing of favourite regex phrases
  • added 'start animations this % in' options to options->media
  • animations will no longer clear to white while scanning beyond their immediate frame cache
  • modifier-click (like shift-click) will now drag animations without altering their pause/play status
  • the splash screen is a little larger, to better fit db update text in
  • the splash screen's text is split into two lines, and shutdown repository processing is a bit more verbose
  • fixed some incorrect dialog text in manage sibling and parent dialogs
  • rewrote sibling and parent pair update code to deal with unusual situations more gracefully
  • improved how the new downloader engine saves error notes
  • the import review frame now lets you copy the selected entries' notes to the clipboard
  • video files will no longer fail to import in windows because of unicode characters
  • recycling files with unicode characters on Linux will work more reliably, and when it breaks, it will fail more gracefully
  • the client and server can now detect and recover from a rare error where a database file is created but never initialised with the appropriate tables
  • the tag/file upload popup message now dismisses itself after a few seconds, rather than pointlessly lingering
  • wx event ids used in one-shot menus will be recycled, which should alleviate the 'after a long time, some menu selections stop doing anything' problem
  • some code cleanup

• version 174

  • 'server stop' and 'server restart' now work!
  • server will now log everything to console as well as server.log
  • server log entries will always be prefixed by the time they were made
  • server shutdown will close services more neatly
  • server shutdown cannot be triggered multiple times with multiple keyboard interrupts
  • server initialisation and shutdown is more verbose
  • server now accepts POST '/shutdown' requests from localhost
  • fixed a server shutdown bug
  • mousewheel events that hit the command and ratings hover windows will be propagated to the media viewer (e.g. if you click a rating, you can now scroll immediately to move to the next file without having to refocus the media viewer window)
  • fixed a bug where zooming out would go immediately to 1% if the media's exact fit zoom was already in the list of default zooms
  • added 'check db integrity' to database maintenance menu
  • multiple drag-and-dropped thumbnails will now be listed inside the drag-and-drop object in their hydrus thumbnail order (was previously random)
  • animations will now pause/play correctly during inbox filtering
  • most client errors now cause a 1 second wait on behalf of the caller, which should bottleneck error spam when it happens
  • fixed some buggy display and rescind behaviour with pre-dialog-existing petitions in the manage parents and siblings dialogs
  • the manage parents and siblings dialogs will now not attempt to remove/petition/rescind a pair if the entered list of pairs has entirely new entries
  • the manage parents and siblings dialogs will now attempt to maintain their sort order after you change some pairs
  • generally cleaned some parent and sibling dialog code
  • added option to options->files and trash for whether to use your OS's recycle bin
  • system:mime text now specifies specific mime group if appropriate, or otherwise lists every mime if >1 selected ('system:mime is specified' is dropped)
  • hydrus repositories will sometimes catch and recover from zero-length update files
  • hydrus repositories will catch processing parsing errors and pause when they happen
  • the new hdd, page of images, and thread watcher import daemons will tidy themselves up if their accompanying pages are completely deleted
  • improved how daemons shut down
  • improved client shutdown logic and speed
  • some misc code cleaning
  • more general code cleanup

• version 173

  • converted import folders to the new system
  • import folders now have names, so you can have several pointing to the same location
  • import folders can filter by mime
  • import folders now have separate actions for successful, redundant, deleted, and failed files
  • these actions can be 'leave alone', 'delete' or 'move to location'
  • import folders now support import file options
  • import folders can now be individually paused
  • import folders' popups are individually optional
  • import folder edit dialog provides a path cache button, like new download pages, to let you review the paths it is remembering
  • existing import folders will be roughly converted to the new system on update, but they will be in a paused state, so you can check they are what you want
  • created new mime selection control
  • mime selection control added to mime system predicate selection dialog, so you can now search by mime more specifically
  • updated some mime system pred code to deal with this change
  • a single non-dragging left-click on an animation now pause/plays!
  • the animation scanbar will now only resume playing the video/flash after a scanning click/drag if it was playing before the drag began.
  • added a 'polite wait' option to options->downloading. this is temporary until I write a better clientside bandwidth throttling system
  • added a 'remove/petition all tags' button to the manage tags dialog to make mistakes easier to clean up
  • added 'confirm send to trash' to options->gui
  • added 'confirm archive/inbox' to options->gui
  • added a simple 'export tags in .txts?' button to the export dialog. In future, I will expand this to do service-specific exports and also add it to regular export folders
  • fixed some invalid name error handling in the manage dialogs
  • fixed a bug that sometimes occured at the end of an admin immediate sync
  • fixed a rare shutdown-order error in video rendering
  • fixed an animation bar initialisation error that was causing error spam on some videos/flash
  • general import code cleanup
  • cleared up admin-side petition wording and presentation
  • improved the threaded workflow and error reporting of the admin-side petition page
  • the thread watcher now reports its number of next seconds until the next check in properly formatted time

• version 172

  • the server no longer needs wx!
  • the server now runs from command line
  • the server is now .py, not .pyw!
  • run 'server help' for more info
  • improved how the controllers start
  • improved how the controllers stop
  • improved controller boot error handling
  • improved how pages close
  • general controller code cleanup
  • general server code cleanup
  • fixed the always_on_shutdown (without asking) option, which was asking anyway
  • removed a debug statement in isalreadyrunning code, whoops!
  • improved isalreadyrunning detection. it should work breddy gud now
  • added gallery file limit to new 'downloading' options page, which folds in the old thread checker options as well
  • added option to always embed the autocomplete dropdown window (rather than having it a floating window), which is now default on for Linux and OS X
  • manage services now supports two kinds of service reset for repositories
  • the service reset buttons now only fire on dialog ok
  • service reset will try, as cpu allows, to update its progress in a message popup window
  • administrators now have a 'sync now' button on the review services window that lets you catch up immediately to the service without having to wait for the normal update time (this will burn cpu time serverside, so be careful!)
  • fixed a bug when searching boorus with unicode-16 characters
  • the client updates directory is neater
  • 'system busy' status is now shown on the status bar
  • 'force unbusy' added to debug menu
  • invalid characters in export filenames will now be replaced by underscores
  • fixed a bug where rating services' cached file counts were not decrementing on de-ratings
  • rating services' cached file counts are reset on update
  • hover windows will pop in over video again, but will not if the mouse is _near_ the animation bar
  • searching for ':' in the autocomplete dropdown will no longer search the db for every single tag jej
  • improved some server_busy logic
  • improved some server shutdown logic

• version 171

  • improved isalreadyrunning code
  • fixed isalreadyrunning code for the Linux frozen build
  • the client's isalreadyrunning splash check will respond more quickly if you choose to wait
  • repository processing sync will no longer update the frontend, which approx doubles its speed
  • repository processing sync now only occurs when the client is idle, so it won't slow your browsing down
  • if you catch the client while it is processing, doing pretty much any action will make it quit out nice and quick
  • repository processing sync now occurs at a lower db level, meaning less laggy popup ui update and overall faster processing time
  • repository popup disappears after a few seconds once it is done--if you leave your client on all the time, you probably won't see it again.
  • split 'maintenance and memory' options panel into 'maintenance and processing' and 'speed and memory'
  • added 'cpu busy' option to options->maintenance and processing
  • added 'run stuff on shutdown' option to options->maintenance and processing to set whether pending db maintenance and repo processing can happen on shutdown
  • added 'max minutes on shutdown' option to options->maintenance and processing to limit how much shutdown processing can be done in one sitting
  • repositories can now sync on shutdown--they will report through the splash window
  • shutdown maintenance is checked and run in a better way
  • moved to a two-stage shutdown procedure
  • daemons are better kept track of, and they talk to the controller in a better way
  • daemons shut down in a better way
  • improved a bunch of idle/busy/just woke from sleep logic
  • improved how the controller maintains the db and memory
  • improved force_idle mode in the client
  • idle calculation is more accurate, and resets on more user-driven events
  • improved responsivity for processing popup
  • fixed a serious bug that was counting some unprocessed content as done when the service processing window was cancelled
  • misc code improvements
  • misc maintenance code improvements
  • fixed a bug with parentless center-positioned dialogs not sizing themselves properly
  • removed delete_orphans timespan param option
  • fixed bugs selecting and media-viewering collected media
  • fixed a missing thumbnail selection bug
  • fixed some os x dialog interaction and z-order problems when the splash screen is open
  • os x splash screen is no longer stay_on_top
  • fixed the 110.00% display bug
  • removed wx from pubsub code
  • pubsubs now route through the controller
  • reduced some pubsub overhead, we'll see if it smooths out some of the gritty stuff
  • misc thread worker pool is now managed by and routed through controller
  • some other thread-wx interaction is now routed through clientcontroller
  • db now interacts with controller directly, not through wx
  • all other indirect references to controller go through hydrusglobals, not wx
  • refactored everything wx away from everything hydrus/server except controller code
  • about window now says the client's network version

• version 170

  • added media viewer mime options to file->options->media page
  • you can now set animated mimes to start in a paused state
  • you can now set the client to 'open externally' any media from a thumbnail activation
  • reworked the way the media viewer generalises and displays its media to obey these new options
  • improved some misc zoom code
  • flv video is now rendered natively by default
  • audio now defaults to an 'open externally' button
  • renamed the nebulous 'url' download page option to 'page of images'
  • moved the 'page of images' download page to the new system--it'll now remember its state through a restart, has a the detailed import status button, all that stuff
  • 'page of images' download page now supports checkboxes to customise which types of links it will search for
  • you can now close and rename page tabs from a right-click menu!
  • deviant art parser now works for mature images
  • improved deviant art tag and image page parsing
  • added 'remove filtered files' option to file->options->files and trash
  • improved UPnP dialog error reporting for when external IP cannot be parsed and for when external IP is reported as 0.0.0.0
  • added an external ip/host override option to file->options->connection page
  • whenever the client or server deletes a regular file or directory, it will now send it to the recycle bin
  • files stored in the db are no longer read-only
  • existing files stored in the db will attempt to be set read&write on update
  • wrapped undo manager access in a lock, which may stop index bugs when a lot of stuff is going on
  • fixed a bug in download progress->progress gauge hook when content length was absent
  • removed previously entered zero-length namespaced tags (like 'wallpaper:') from client and server
  • moved 'client/server already running' check from db access test to more reliable process list review
  • improved 'client is already running' mini-dialog
  • added cpu percent utilisation check (if any cores >50%, dump out) to idle check
  • if the 'canvas zoom' value exactly matches a default zoom, it will not be inserted into the list of zooms to scroll through
  • the 'canvas zoom' will be displayed with hundreds of a percent accuracy
  • the new thread and hdd import pages will now obey page-global pause/resume events
  • the new thread and hdd import pages now have the 'sort by' dropdown
  • the new thread import page supports set_search_focus event
  • added import tag options info to 'getting started - tags' help page
  • relabelled the path tagging button on the import files dialog to something that made more sense
  • might have patched an unusual subscription error reporting bug
  • updated sqlite on windows--db should be a bit faster
  • updated pillow on all platforms
  • compiled a new python for my os x build with newer openssl library, which should fix some ssl problems people were having
  • updated some deprecated directory parsing code
  • general code cleanup

• version 169

  • revised all the help files
  • updated out of date screenshots in the help files
  • added some more screenshots to the help's index
  • cleaned out unused help images
  • fixed some missing service error catching in the custom filter setup dialog
  • fixed random reordering of paths from import files selection dialog
  • logged errors should now be prefixed by the time they occured
  • errors should now be more reliably printed to the log
  • fixed a problem with e621 queries that included '/'
  • improved some server temp file error handling
  • fixed a typo that was throwing errors when trash settings were set to 'no limit'

• version 168

  • thread watcher is moved to the new system--it will remember its previous state
  • thread watcher has more compact, flexible gui layout
  • thread watcher now supports file import status button, so failures can be reattempted
  • thread watcher check now button has improved logic
  • import tag options and import file options gui controls now plug into the new system
  • thread watcher will now remember its tag and file options
  • thread watcher will accept changes to its tag and file options after it is started
  • fixed a bug in import tag options deserialisation
  • hdd and thread watcher static texts will flicker less
  • improved some network transfer gui reporting code
  • send to trash, delete from trash, and undelete now all have different icons in the command hover window
  • added open externally button to command hover window
  • opening an animated gif or movie externally from the media viewer will pause it in the client
  • share->copy->image added to media viewer for static images
  • fixed a typo bug in copy 'image' to clipboard
  • fixed a typo in external ip discovery that was causing errors for local booru external link generation
  • improved external ip discovery error recovery
  • in prep for the de-wxing of the server, refactored a lot of code so the server and client code don't import each other at any point
  • if an attempt to close the client is aborted (e.g. because of an active import page), the exit splash screen will now destroy itself

• version 167

  • created gui control to show file import status
  • added a button to launch this control, wrapped in its own frame, to the hdd import page
  • you can now drag and drop thumbnails out of the client for quick export!
  • the client will catch internal drag and drops, and won't interpret that as an import request
  • dragging files onto the client will now always show the 'copy' mouse icon
  • some thumbnail generation code is a bit quicker--resized thumbs are now always generated on their first viewing
  • if you use 200x200 thumbnails, the client won't bother to generate resized thumbs, saving space and time
  • the text entry dialog will now check its button status on a non-keyboard related text-change (e.g. selecting paste from the right-click menu)
  • the popup created by downloading a raw url or a youtube video will finish itself properly when the import attempt is completed, removing the pause and cancel buttons and making the popup dismissable
  • the splash screen will now stay on top
  • the password entry dialog will also stay on top
  • you can now copy the md5, sha1, or sha512 hashes of local files from the normal share->copy->hash menu
  • fixed an overly broad deleted/redundant test on md5/url pre-import status checks
  • options to remember size of manage tags and position of both manage tags and ratings dialogs added to file->options->gui
  • updated some of the help

• version 166

  • created new object to hold hdd import information
  • created new object to hold generalised import file status
  • moved hdd management controller to the new system
  • moved hdd management gui to the new system
  • hdd imports will now remember their import and pause status through a session change
  • misc import code improvements
  • hydrus client is getting out of the zip business--zips are no longer parsed for import nor able to be created for export
  • import and export code is simpler
  • repository file downloading daemon will now throw up a small auto-dismissing popup when it downloads files
  • repository file downloading daemon will respond to new downloads much quicker than previously
  • if a repsitory is in the process of backing up when it receives a request, it will now return a 503 'server temporarily busy' error (rather than timing out)
  • the client can now catch and handle these 503 errors gracefully
  • backing up will no longer block the client's gui, and it will no longer timeout if the operation takes more than ten minutes
  • some networking code is a little simpler
  • added 'select local files' and 'select trash' to thumbnail right click menu
  • 'cancel download' added to file repository submenu of thumbnail right click menu
  • some buggy listbox resize behaviour (scrollbars not disappearing and occasional layout and drawing update fail) has been fixed

• version 165

  • added a db table to track when a file was sent to trash
  • added an option for whether trashed files are removed from view
  • added a max age to trashed files
  • added a max size to the trash
  • added a daemon to maintain the trash
  • improved some generic deleted shortcut logic, so delete key will present option to delete from trash if appropriate
  • shift+delete will now undelete
  • misc trash code improvements
  • thumbnails deleted from the trash will accurately remove their inbox status icons
  • images downloaded from file repositories will accurately add inbox status icons
  • reduced chance of problems when trying to delete media from the trash when it is currently being rendered
  • further reduced this chance
  • removed redundant undelete code, folded it into add/delete code
  • the media viewer will now remove files deleted from the trash in all cases, even when launched from a file repository search
  • significantly improved how animations figure out when to put frames on screen. these timings should be much more accurate, slowing down only if your CPU can't keep up
  • 8chan thread watcher now parses all files from multiple file posts
  • if a booru provides a link starting with 'Download PNG', which refers to the original source png, that will be preferred over the jpg (konachan and yande.re, which run Moebooru, do this)
  • booru parser is now a little more liberal in parsing for the image link
  • yande.re booru support is added by default
  • fixed some local/remote state code, which was breaking file repository searches in several ways
  • improved error handling in repository file download daemon
  • cleaned up manage options dialog code
  • reduced min size of the media caches in manage options dialog
  • moved thumbnail size to 'maintenance and memory'
  • added better error handling to repositories that cannot find an update file during processing
  • repositories that have such errors will pause to give the user a chance to figure out a solution
  • misc code improvements
  • fixed a bug where file repository downloads were not importing if the file had once been deleted
  • dropped the clever indices from last week--sqlite had too much trouble with them. I will reform my db tables next week to get around the whole issue. for now, I have reintroduced the bulky index and forced sqlite to use it where appropriate
  • added a test for collecting pending tag data
  • tags in the form 'text:', which would sometimes slip through when typing quickly, are now not processed
  • improved tag cleaning error reporting
  • improved when special wildcard and namespace predicates are provided
  • namespace predicates now display as 'namespace:*anything*'
  • fixed a bug when launching the media viewer from collected results
  • fixed a command hover window layout bug that was putting namespace text in incorrect places
  • fixed a bug that was causing the new client-wide processing phase option not to work because, ironically, its calculation was out of phase
  • review services will now state how many updates have been processed
  • review services will specify if an update is imminently due, rather than saying the repo is fully synched
  • fixed a review services layout bug that was misaligning text after an account or data usage update
  • fixed a bug in system:similar_to serialisation, so pages with this predicate can now be saved to a session
  • fixed the same bug for system:hash
  • vacuum should be a bit politer about waiting for its popup message to appear
  • database maintenance won't run while a repository is synchronising

• version 164

  • rewrote the drawing code for the listbox that displays tags in various ways to be a lot faster and more memory efficient
  • updated one new client mapping index that wasn't working quite as I wanted it to something more clever
  • db will be a little smaller and mappings stuff will be even faster
  • merged the two ratings system predicate input panels, so you can now select like/dislike and numerical ratings system predicates at the same time
  • fixed booru download page serialisation, which means they will save to sessions
  • prototyped trash service
  • locally deleted files will now be sent to trash
  • locally deleted files will not be removed from the existing search
  • files can be permanently deleted from trash, which will also immediately physically delete them from your hdd
  • files can be restored from trash back to the local file service
  • inbox state is now more separate from the local file service, so it will be remembered through a visit to the trash
  • improved delete code all around
  • general inbox/archive db code improvements
  • misc content update pipeline improvements
  • optimised mass-adding of files to a service (for instance, when (un)deleting a whole bunch of files!)
  • delete orphans daemon is removed--it will be replaced by a more thorough single-shot hdd/db purge like 'check file integrity'
  • files are not yet automatically removed from the trash--I will add that next week.
  • updated db access info in db folder
  • added sqlite command line executable to db folder for all platforms
  • bit of code cleaning
  • cleaned up some gui error reporting
  • might have fixed a service cache bug in the db that was causing double bandwidth reports and possible looping sync behaviour

• version 163

  • reconfigured some important mapping indices in the client db to reduce search time for many common tag operations
  • the new indices have also sped up tag processing significantly
  • added an automatic db optimiser analyzer run whenever the db updates
  • pixiv now downloads the largest version of an image again
  • pixiv tag parsing improved
  • added support for some unusual mp4 types
  • the noquery media panel is removed. pages with no query status will now show a normal thumbnail panel with 0 files, rather than 'None' files.
  • check file integrity will now report number of missing files and incorrect files separately
  • check file integrity can now take a folder to move incorrect files, rather than deleting them
  • when you try to pend a tag with a sibling in the manage tags dialog, you will be prompted with the chance to pend the sibling instead
  • the rating system predicate dialog now allows for easier (and multiple!) selection of like rating parameters
  • the rating system predicate dialog now allows for easier (and multiple!) selection of numerical rating parameters
  • the rating system predicate dialog now offers correct predicates for 'zero allowed' numerical rating services
  • some bmp icon code cleanup
  • changed pause and cancel buttons to bitmap buttons
  • cleaned up a lot of pause and cancel code
  • now, when popups are paused, they will hide their details
  • popups that are cancelled by a dialog change (for instance when subscriptions are changed while subs are processing) will now dismiss themselves after a few seconds
  • during repository sync, http failures are recovered from more gracefully, and any pending processing will continue
  • the repository update downloading and processing loops will wait on the db and gui more efficiently
  • fixed a bug in the way the manage export folders dialog was testing existence of subdirectories
  • general code cleanup
  • bit more code cleanup
  • some good daemon refactoring

• version 162

  • on update, all previous gui sessions are deleted!
  • on update, all export folders are deleted!
  • made an important efficiency improvement to the new serialisation protocol--network version is incremented as a result
  • finished extracting management data from management panel
  • management data is wrapped in management controller, a new JSONable class
  • pages collapsed to a single class
  • page and management panel instantiation completely rewritten for the new system
  • created new JSONable guisession class for better session management
  • db is updated to store all this stuff in JSON rather than YAML
  • session loading is much less bloaty for clients with many sessions
  • all page types are now serialisable and hence addable to a session
  • moved predicates to JSON
  • moved filesearchcontext to JSON
  • autocomplete dropdowns now remember their tag service, include current, include pending and synchronised status over sessions
  • export folders now remember their file and tag services, and also include current and pending status
  • some general improvements to export folder code
  • cleanup of export folder dialog code
  • added a test for export folders
  • some general refactoring
  • some general cleaning
  • significant refactoring of predicate variables
  • fixed a bug in multi-version update from before v154
  • if you do not have any ratings services, the 'manage->tags/ratings' submenu will now just be 'manage tags'
  • changed some thumbnail menu entries to say 'selected files' instead of 'all files', which is more accurate, see if you like it

• version 161

  • updated windows python, sqlite and ffmpeg
  • added linux ffmpeg binary to the executable build
  • updated os x ffmpeg binary
  • animation is generally working for linux and os x
  • fixed some linux/os x ffmpeg calls
  • fixed webm import for linux/os x
  • webms work great
  • moved to rendering gifs with PIL by default on linux and os x, which has bad palette support but renders more reliably
  • the animation scanbar position caret is now visible on linux and os x
  • fixed some timing/resume issues the animation scanbar on os x
  • the old mediactrl video embed is removed
  • mp4, wmv, mkv are added to native rendering control, no audio yet (just like webm)
  • flv and flash will now have a little vertical padding on max zoom to make vertical hover windows easier to access
  • changed database to a faster synchronisation mode
  • tuned update processing daemon for more accurate time calculations and faster baseline speed, even if that knocks gui latency a bit
  • fixed rows/s being thrown wildly off by long pauses
  • made update processing daemon more reactive
  • improved some of the flow and generally cleaned the update processing daemon
  • added processing phase option to regular delay to processing of updates (this is useful if you run multiple clients and don't want them to process at the same time)
  • fixed e621 tag parsing
  • completely rewrote the focus engine behind the autocomplete dropdown
  • fixed a bunch of autocomplete dropdown's generally buggy behaviour for linux and os x
  • fixed autocomplete dropdown's display, sizing, and positioning for linux and os x
  • fixed autocomplete dropdown selecting system predicates with the keyboard
  • fixed autocomplete dropdown show status in linux with multiple pages open
  • improved all subprocess calls, removing interim shell step and parameterising passed arguments
  • improved how files and directories are launched in windows
  • upnp is fixed for linux
  • fixed hover window archive/inbox/delete icon buttons, which were spamming their commands to all open media viewers, not just their own
  • added a simple raised border to hover windows to better delineate them from the canvas background
  • updated the hover window size and position code to be a bit more reliable (still seems to bug out a bit on linux)
  • tags hover window should now expand sideways more reliably
  • it is now not possible to create nested export folders
  • searching for numtags < x will now include files with no tags
  • fixed manage boorus dialog OK in linux
  • fixed booru selection mini-dialog in linux
  • generally improved restoring database code
  • fixed restoring a database when you have tag archives in existing database
  • fixed a unicode encoding error when converting certain jobs to text (this was throwing errors in deviant art downloads/subscriptions--we'll see what was actually going on now, whether this is an error not being formatted right or something else)
  • started some great rewrite of management panel gui code
  • rearranged the download panel gui hierarchy slightly
  • improved collapsible panel collapse/expand layout code
  • removed code and database table for the old numerical ratings filter
  • some general code refactoring and cleanup
  • improved client upnp daemon timing
  • harmonised how the client and server check requests for bandwidth-tracking eligibility

• version 160

  • added options for http, socks4, socks5 proxying
  • improved some network-related errors
  • ratings services can have custom border and fill colours for their various states
  • ratings services can now also be squares or stars instead of circles
  • numerical ratings services can disallow zero ratings
  • fixed the JSON parsing error that broke the thread checker
  • shrank the width of the thread checker to make it a little less ugly
  • the autocomplete tag search dropdowns will now accept and search with quickly entered text
  • this new system will substitute siblings in the manage tags dialog
  • fixed rows/s average calculation
  • rows/s is more accurate
  • content update popup string update is less laggy
  • content update popup now shows content rows, rather than content parts
  • removed update 'taking a break' component, as it was not doing the job I wanted it to
  • db debug profile mode can now be turned off lol
  • fixed an error from middle-clicking greyspace in the linux notebook tab area
  • general code cleanup
  • some string conversion code cleanup
  • fixed a missing canvas bmp error with flash/flv embed buttons

• version 159

  • split previously monolithic repository updates into smaller pieces
  • added service_update server calls
  • extended content_update server calls to support sub-indices
  • sped up some content update preprocessing code
  • improved some content update preprocessing code
  • radically reduced serverside memory usage while generating updates
  • added iterator splitters to make sure any single update row cannot be too large
  • thanks to iterator splitter, updates should process through the client more smoothly
  • added timespan splitter to make sure any single server update query cannot be too large
  • content updates are resumable if broken part way through downloading the list of them
  • the update popup will state how fast it is currently working in rows per second committed
  • cleanup of a lot of update related code
  • more cleanup of update related code
  • improved serialisable protocol so it'll work better over a network
  • made serialisable protocol much simpler
  • fixed numerical rating system predicate dialog slider range
  • fixed numerical rating system predicate dialog OK for valued predicates
  • fixed numerical rating custom filter action dialog
  • improved some network yaml error handling
  • replaced 'export tags' thumbnail menu entry with a tag archives system
  • replaced 'import metadata' file menu entry with a tag archives system
  • disabled thread dumper and manage imageboards and manage 4chan pass dialogs for now, because dumping code is out of date and completely screwed up
  • client will now not start in idle mode
  • help debug menu has new 'force idle mode' entry
  • idle mode is displayed on status bar
  • simplified client main gui status bar display and code
  • the flash and flv files' embed button now has a little border to delineate it from the canvas background
  • fixed some clientside bandwidth tracking code
  • removed some old networking code
  • made some custom objects draw themselves more efficiently
  • for now, manage tag parents and siblings dialogs will not show deleted rows. I will eventually add a 'show deleted' checkbox, like the manage tags dialog has
  • some static image rendering is slightly less laggy, particularly when browsing large images
  • fixed initial height of manage tags dialog launched from preview window
  • changed the 'search area' vs 'preview' sash gravity so the preview area won't expand or shrink on resize--see if you like it
  • autocomplete dropdown should now hide itself when focus goes to other programs
  • autocomplete dropdown should now hide itself when focus goes to other hydrus frames in os x
  • fleshed out new URLCache object to handle better gallery download url management
  • refactored some file status tracking stuff to a better system
  • refactored gallery page fetching to a better system
  • refactored gallery url handling to a better system
  • some redundant import checking should be much faster
  • improved error handling when booru image page parsing fails to find image url
  • refactored how tags are fetched for DA, tumblr, and giphy as part of the above overhaul
  • misc code cleanup

• version 158

  • subscriptions now have a 'get tags even if file already in db' checkbox, defaulting to false, which will significantly speed up subs with redundant files
  • subscriptions will now more accurately obey initial limit
  • gallery downloaders will now more accurately obey file limit
  • merged some hentai foundry code into the general downloader
  • cleaned and refactored a lot of import page initialisation code
  • cleaned and refactored of lot of import management panel code
  • cleaned and refactored of lot of import code
  • created numerical rating control
  • added numerical rating control to rating dialog
  • added numerical rating control to rating hover window
  • updated like and numerical rating service info for future support of custom colours and shapes
  • fixed listbooks page deletion display
  • fixed manage boorus saving changes to existing panels
  • fixed manage imageboards saving changes to existing panels
  • fixed manage imageboards saving remove imageboard
  • improved and refactored listbook code, further separating active and proto pages
  • improved some listbook name conflict error handling
  • misc cleaning
  • export folders and the export dialog will also export file attributes like access and modification time
  • export folders will now overwrite destination files if they are a different size to the exportee
  • export folders can now be set to 'synchronise', and so will delete any other files in the directory. existing export folders will behave exactly as before
  • improved misc export and import folder code
  • moved folder type tooltip to an explicit bit of text on export and import folder edit dialogs
  • thumbnails can now display multiple hydrus file repository icons (current, pending, and petitioned) if multiple states apply
  • hydrus file repository icons are gone from media viewer--now there is a text list
  • hydrus file repository text list is added to hover window as well
  • exclude_deleted option will no longer hide files from file repository search results--it was more confusing than helpful
  • improved some error-prone logic in how advanced_import_options were being stored
  • ratings hover window icon background colour is fixed
  • imported bmps should now be converted to pngs with 1:1 colour (before, they were being collapsed to a 256-colour palette, which was showing ugly dither on complicated images)
  • fixed a sometimes segfault crash on search refresh on OS X
  • the preview window now supports a simple right-click menu
  • added 'open installation folder' to file menu
  • improved some media viewer precaching code, which should speed up some scrolling, particularly on the first scroll
  • improved some layout code
  • improved some file permissions code
  • cleaned some canvas code

• version 157

  • fixed a bug in listbook page initialisation and retrieval that was affecting many dialogs on OK
  • some general dialog OK code cleanup
  • fixed a media-navigation bug in managetags dialog
  • fixed a serious OK bug in imageboards dialog
  • created a new 'periodic' object to manage future subscriptions timing improvements
  • started subscription YAML->JSON conversion
  • stopped compressing json in the client db so it is human readable and thus easily editable
  • subscriptions are no longer renamable, as this code was super buggy and could delete other subs
  • tidied up the database menu
  • a bit of misc cleanup
  • in many cases where it was once pseudorandom, services are now reported in alphabetical order
  • prototyped 'like' ratings control
  • added new like ratings control to the background bmp of the media viewer
  • added new like ratings control to the manage ratings dialog
  • added new like ratings control to a new hover window in the top-right
  • added basic additional icon support to new hover window
  • fixed some misc new alignment bugs related to new ratings stuff
  • like ratings controls on the hover window have tooltips
  • fixed up some icon/rating display logic in the background bmp of the media viewer
  • updated ratings dialog error handling

• version 156

  • improved my build workflow in several ways, which should result in fewer orphan files and other build weirdnesses
  • some bad path usage in initialisation icon resizing has been moved to better temp paths
  • hitting page up or down on a manage tags dialog launched from the thumbnail grid no longer clears the current media
  • improved file permissions code across the program
  • fixed import folders daemon's test code for non-windows
  • fixed up some temporary file code that wasn't cleaning up those files when the application was about to close
  • fixed a newline parsing problem in copy/paste tags in the manage tags dialog
  • added tag cleaning to pasting in the manage tags dialog
  • added newline removal into standard tag cleaning process
  • fixed a server db bug that was stopping some accounts from being created
  • fixed some network session exception creation and catching
  • new popup messages should no longer steal focus in most circumstances
  • client should recover from serious popup message manager errors better
  • hover windows will now only pop up if their media viewer is the currently focused frame
  • hover windows will not hide until the mouse moves off them when flash or webm are underneath them
  • os x will no longer vanish media in the media viewer on an action like archive or inbox
  • fixed juddery media mouse dragging in linux
  • improved the way listbooks work to avoid a problem with clientdata in wx linux
  • export folder is gone as the default export location--now it is 'hydrus_export' under the current user's home dir
  • updated windows ffmpeg to latest version
  • fixed an important opencv dll conflict that was causing some gifs to render wrong in windows
  • shift focus media logic improved--shift initial thumbnail is now last image selected
  • shift selection will no longer deselect anything

• version 155

  • fixed a frame seek error when looping long and/or large gifs with unusual palettes
  • improved recovery when fps reads as 0 on videos
  • added error detection and graceful recovery and reporting for missing ActiveX flash control
  • removed a lot of old imported messaging code that was slowing down boot
  • removed some other old library imports that are no longer needed
  • gracefully silenced 'application shutdown' daemon errors (they were previously spamming to console)
  • fixed a hover window display check that was sometimes spamming linux with edge-case errors
  • slideshows will now pause while the right-click menu is open, stopping bizarre full program crashes occuring on many menu items while slideshow was in progress
  • fixed a thread_id-sqlite-cache_initialisation problem that was breaking tag archive sync
  • harmonised copy/paste tag protocol in manage tags dialog with 'copy all tags' of the tags list
  • removed some old messaging code in client db creation

• version 154a

  • fixed a bug in v154 update code when there was more than one set of shortcuts to convert
  • fixed a faulty default value for num_pixels system predicate that was stopping options from opening
  • fixed an error when video fps is 0

• version 154

  • managed to completely break my linux and os x dev environments trying to update python--their release will come soon, once I've cleaned them up
  • removed some old cv code
  • updated to opencv 2.4.11 on windows
  • fixed a bug in trying to upload a small number of tag petitions
  • hover command buttons are now shrunk to exact fit size
  • added zoom buttons to command button hover window
  • added navigation buttons to command button hover window
  • added fullscreen switch button to command button hover window
  • added archive/inbox button to command button hover window
  • added delete button to command button hover window
  • added generic close window button to command button hover window
  • added tooltips to all the command buttons
  • cleaned some misc canvas code
  • hover windows now fit better, without overlap
  • moved media info strings to top of media canvas, so they hide behind hover header
  • hover windows will now not cover webms or gifs when the mouse is over the media container
  • hover command buttons now work for inbox filter, including back/skip buttons
  • inbox/archive button feels and works correctly for the inbox filter
  • updated manage options dialog's 'default file system predicates' mess to use the new cleaner predicate panels
  • all file system predicate defaults have been reset to default
  • added system:num_pixels to file system predicates default panel
  • fixed the system:num_pixels predicate being broken on options save
  • removed the ratings system predicates from file system predicates default panel
  • fixed misc system predicate bugs

  • moved yaml->json conversion forward

  • settled on object code
  • added built-in compression to serialisation
  • added db tables and access code to support it
  • fixed a little layout mess in the edit custom filter action dialog
  • created a rich 'shortcuts' class for storing shortcut->action information that can be easily expanded to handle mouse events as well
  • new shortcuts class works on new json storage rather than yaml
  • moved custom filter shortcut action storage and general handling to new shortcuts object
  • moved old favourite_shortcuts tests to new code
  • cleaned a lot of the custom filter dialog code
  • rewrote the custom filter setup dialog to work more like other dialogs--now all changes are saved on ok, and save/save as/delete is replaced with add/delete
  • auto-creation of 'previous' shortcuts is removed, as all changes are saved to shortcut sets anyway
  • removed the ratings filters--the numerical filter may make a return in another form, your thoughts would be appreciated
  • fixed a cache counting bug when archiving redundant files during import
  • sped up file deletion a little bit
  • added an inbox cache to reduce laggy inbox checking, particularly search result fetching on dbs with large inboxes
  • fixed subscriptions that have no initial file limit, please reset your sub url caches to fix in these cases
  • did a lot of server/client database merging and refactoring
  • improved some database error handling
  • fixed an error in file repository superbans
  • optimised some critical db code
  • sped up tag censorship filtering
  • added a db profiling mode, accessed from the help menu, that will dump copious db profiling info to the log
  • misc code cleaning
  • improved some popup mesage print sync timing
  • rewrote some server auto-setup code to deal with slowly-starting server

• version 153

  • cut out some out of date stuff from help
  • put a nice big red warning at the top of the 'running your own server' page
  • added system:num_pixels for megapixel searching
  • system predicates height, width, ratio, and the new num_pixels are now collapsed to a super-predicate, system:dimensions
  • system:rating will now only show if you have some ratings services
  • the system:rating dialog will now hide like or numerical ratings if you have no services of that kind
  • complete refactorisation of system predicate dialog
  • some misc system predicate bugs and bad panel event precedence fixed
  • shortened all predicate spinctrls
  • improved how system predicates are stored and used for a search
  • moved some view menu items to the download menu and the new search menu
  • all of the download gallery pages are added to the download menu
  • added some explanatory text to the pixiv dialog
  • the tag uploading process will more carefully prepare its subupdates so that mass allocations of the same tag to many files will be split into smaller chunks, avoiding connection timeouts in these cases
  • hover windows will no longer hover over flash files
  • tag hover frame will resize a little based on the canvas width
  • the twenty pixel mouse warping when the mouse starts dragging close to the edge of the media viewer will no longer occur on flash files
  • the canvas buffer for flash files is expanded to five pixels either side
  • the hover tags window will now update itself when the media's tags change
  • new hover window for details and commands, up top of the media viewer
  • moved the old popout windows' buttons to the new hover window
  • added hover window commands for the normal browser
  • cleaned up some general canvas code
  • index string will now display with a slash in all cases (it was a backslash for windows, for accidental reasons)
  • file limit for subscriptions and download pages now has minimum value 1
  • searching for number of tags while tag censorship has some namespaces will no longer throw an error
  • moved boot and exit code responsibility from the splash frame to the controller
  • improved the boot and exit code generally, including dragging feedback lag and error handling
  • improved some thread error handling
  • fattening service info won't trigger on client shutdown quite so much, which should speed up shutdown a little

• version 152

  • added prototype hover frame for tags in media view
  • hover frame shouldn't show when a dialog is open
  • manage tags dialog launched from media viewer now has a delete button
  • subscriptions now have an 'initial limit' variable, defaulting to 500, that limits the total number of files the subscription daemon will look for on the initial sync
  • added a similar file limit spinctrl to gallery download pages
  • updated layout of import files dialog
  • cleaned up some ugly global variable scope stuff
  • fixed initialisation of advanced import options in file import dialog
  • made a good start to better object serialisation
  • subscription and repository sync daemon jobs that stop due to a dialog-driven change during processing will now cancel themselves after five seconds
  • fixed namespace (e.g. 'series:') tag censorship
  • fixed the fullscreen switch bug that was breaking an (initially fullscreen)->(non fullscreen) media viewer
  • fixed some search logic (some system predicates were not firing when there were no regular tags present)
  • removed some artificial delays on daemon db access, let's see if it chokes anything
  • harmonised a bunch of client and server controller code
  • created a common controller class and merged a lot of the client and server controller code into it
  • general code cleanup
  • more general code cleanup

• version 151

  • added a possible solution to the manage tags dialog next/previous buttons crash
  • completely overhauled temporary file management throughout client and server
  • removed old temp folder
  • copy files to clipboard will now copy the database's exact file paths (rather than copying to a temp folder beforehand), making it a lot quicker
  • the import folders daemon will no longer attempt to import (and hence spam errors about) zero-length files
  • added one-time tag archive sync button to the 'perform a service-wide operation' dialog in review services. this lets you add or delete the archive's tags by namespace
  • revised the tag archive sync code a little so it syncs to non-local files as well, if the hashes can be lined up
  • added hash cross-referencing to archive sync code before maina processing step, often saving a huge amount of time
  • fixed a parsing error with advanced content updates at the gui-end
  • added GetName and IterateHashes to hydrus tag archives
  • added default 'similar to' max hamming distance to file system predicate defaults in client options
  • fixed up the default text display of the file service system predicate
  • added some text to the sort/collect options panel to better explain what the big listbox is for
  • fixed a popup numerical typo when uploading files to a repository
  • fixed an utterly bizarre bug caused by the import files dialog that was screwing up subsequent dialogs' layout and size, apparently by wx voodoo
  • refactored all the functions and classes of hydrusconstants to better locations
  • misc other refactorisation
  • misc cleaning
  • renamed hydrusdownloading to clientdownloading
  • changed all default { } and [ ] parameters to None. I don't think it mattered, but just in case for the future
  • moved some ugly global variable stuff to better places, hoping to do more in future
  • cleaned up many unused imports

• version 150

  • added an 'idle' updating processing mode that churns through tags four times faster but makes gui laggy. it will kick in whenever you haven't done anything in thirty minutes
  • improved some gallery downloader status texts
  • gallery downloader will no longer wait 5 secs before fetching first page of urls
  • fixed a critical race condition that had a chance of causing complete and unrecoverable gui freeze in the gallery downloader code
  • fixed rendering of some bizarro pngs that CV couldn't understand
  • fixed tag parent management for admin users
  • fixed up some tag parent dialog code
  • tag sibling dialog now accepts single reasons for multiple pairs, like the tag parent dialog
  • improved error state on rendering video that reports resolution of 0x0
  • removed looping error reporting on certain static image rendering errors
  • fixed an error when some thumbs were 'collected' while fading in
  • tagboxes now select on home/end and support numpad versions of page up/down
  • fixed a radio button initialisation bug
  • boosted the initial size of the 'set default advanced tag options' dialog so the collapsible pane is more visibile
  • silenced a upnp error when external ip can't be parsed
  • reduced some tag dialog previous/next button lag
  • general code cleanup and refactorisation
  • gui flag refactorisation and unification
  • refactorisation of most non-constant stuff from constants files
  • refactorisations of media mixins class in prep for a general rewrite of that code
  • improved client initialisation of default options, boorus, and imageboards

• version 149

  • all listctrls with an attendant delete button will now support the same action with the delete key
  • several normal tag boxes will support the delete key as well
  • added a temporary service cache to the client db to speed up service fetching until I can figure out a yaml replacement
  • cleared out inefficient tag and hash database fetching and generation code
  • tag parent dialog will now ask for an action/reason for every ( children, parent ) pair rather than every ( child, parent ) pair
  • fixed an error with collecting by rating
  • fixed the 'fix' to the gui_colours db update problem
  • the autocomplete dropdowns in the paths-to-tags dialog will no longer produce errors when tags are submitted
  • improved accuracy of taglist scroll-to-selected scrolldown calculation
  • collapsed some of the helter-skelter db code in prep for future improvements
  • moved daemon code to separate files
  • silenced server upnp errors
  • all except: continue lines are banished!
  • cleaned up some miscellaneous exception code
  • cleaned some ugly behind the scenes a/c dropdown button code
  • did some general code cleanup

• version 148

  • rewrote thumbnail canvas and scrolling code from a crashtastic monolithic bmp to a lighter, faster, and more flexible page buffer
  • all custom gui elements should be less flickery
  • the manage tags dialog will now grow significantly taller if its parent window is also tall
  • dialogs launched by the media viewer will initially position themselves according to that, rather than the main gui
  • dialogs launched by controls will initially position themselves according the control's toplevelwindow, as originally intended
  • thumbnail download (for file repositories) no longer happens silently in the background; it will now occur in the repository sync daemon, reporting its progress in the normal repo sync popup
  • missing thumbnails are now replaced with the hydrus psi symbol silently, with a simple statement written to the log
  • fixed a tiny typo in update error code that was reporting version wrong
  • fixed a typo bug in gettagarchivetags
  • fixed a typo that had broken namespace sorting
  • numerous other single-line miscellaneous bug fixes
  • fixed a bug with displaying media with size (0,0)
  • fixed a bug with zooming in flash files
  • improved some buggy tag selection logic that was sometimes desyncing indices between menu popup and selection
  • tags will now stay selected even through changes to the tags list
  • any attempt to close the autocomplete dropdown floating frame should now bump the close event up to the whole program
  • linux release now includes source code alongside executables

• version 147

  • fixed a problem when trying to do a multi-release update that contained the v146 update
  • fixed the 'canvas not resizing after media removal' bug
  • when an autocomplete appears in a dialog, the dropdown window will integrate into the dialog (rather than being a popup), allowing mouse interaction
  • refitted the paths-to-tags dialog so the different static boxes and newly embedded A/C dropdowns fit better
  • fixes to and cleanup of sibling tag retrieval code
  • fixes to and cleanup of sibling tag filter code
  • fixes to and cleanup of sibling tag search code
  • fixes to and cleanup of sibling tag display code
  • fixed a couple bugs in READ autocomplete tag search caching that was stopping namespaced queries searching properly if typed in manually rather than pasted
  • fixed a similar bug in WRITE autocomplete tag search
  • fixed a bug that was ignoring namespace entirely in WRITE autocomplete
  • synchronised subscription popups' new file buttons with the text

• version 146

  • manage tags and ratings dialogs will now OK on F3/F4, not CANCEL
  • zoom switch (default shortcut 'z') will now work for images smaller than the media viewer's canvas
  • in the media viewer, the canvas-fitting zoom value is now inserted into a media's scrollable zooms
  • added 'tags box background' gui custom colour
  • volume, chapter, and page tags will sort properly again in thumbnail view, including last week's improvements
  • the thread watcher will no longer break its checking loop on non-404 http request or parse errors (i.e. the manual 'check now' button will still work after a misc error)
  • to remove clutter, the preview window will no longer show tags and file information like the normal media viewer
  • subscriptions will now show a live 'show files' button as they process
  • the copy and 'open new page' right-click menu options on tags will now work on more controls
  • cleaned quite a bit of tags box code
  • cleaned up and unified a lot of miscellaneous canvas zoom and display code
  • fixed an error when trying to upload a file petition without any accompanying file uploads

• version 145

  • added custom gui colours for thumbnail backgrounds and borders, the autocomplete background, and media background and text
  • added <- and -> arrows to manage tags dialog launched from navigable media viewer
  • on empty input in the manage tags dialog, page up and page down work as shortcuts for the new <- and -> buttons
  • fixed the media height calculation for animations, so when they are vertically scaled, the total height including scanbar won't overflow off screen
  • allowed non-integer page, chapter, volume tags in display and sort calculations
  • semi-integer tags will sort along with integer tags and string tags like so: 0 < 0a < 0b < 1 < 2 < 22
  • improved the old tag/media sorting code
  • removed loli and shota from hentai foundry filter options
  • patched old db-stored predicates to attempt to convert to the new format when queried for _inclusive
  • this _should_ have fixed the recent export folder problems
  • created an 8chan board, and updated my various links, including in the client, to migrate from my old forum to this
  • added a 'fit to canvas' checkbox in file->options->media that will zoom small images to the full size of the media viewer
  • misc code improvements

• version 144

  • files named 'Thumbs.db' will now be skipped in the import files dialog
  • fixed wildcard searches, which last week's predicate rewrite broke
  • fixed a typo that was showing namespace predicates as exclusive (-series:*) when they were actually inclusive (series:*) and vice versa
  • added wildcard namespace searching for database autocomplete queries
  • fixed database wildcard autocomplete searching when wildcard match is not the first word in a tag
  • fixed database file searching when wildcard match is not the first word in a tag
  • added a comprehensive suite of predicate-unicode conversion tests
  • cleaned and improved some of the downloader code
  • added five second per-gallery-page delay to subscription daemon
  • added three second per-file-delay for regular gallery downloads and subscriptions, just to be polite to those web services
  • added three second per-file-delay to the thread watcher for the same reason
  • added a 'check now' button to the thread watcher
  • fixed a problem that was sometimes causing subscriptions, when paused, to continually restart
  • removed unnamespaced tag parsing from deviant art
  • fixed creator parsing for deviant art, which was formerly cutting off the first character
  • patched an account sync problem in the manage tags dialog
  • in add tags by path dialog, tags are now sorted before being added to the file list
  • in add tags by path dialog, the regex sections now generate tags for every match in the string, not just the first
  • stopped collapsible panels resizing dialogs to minsize on collapse or expand
  • added shortcut for 'open externally', default Ctrl+E
  • moved 8chan to new 8ch.net domain (old domain still works)

• version 143

  • when making a READ autocomplete tag query, instances of tags that only have a count in a single namespaced domain will no longer accumulate helper results in the non-namespaced domain i.e. no more 'blah (1)' 'title:blah (1)' dupes
  • improved the way the above results are calculated
  • pixiv artist downloading now correctly asks for numerical artist id
  • reworded some of getting started with files help page to better explain multiple selection
  • widened the splash screen a bit so db update messages have more space
  • all frames and dialogs now have an explicit minimum size that is usually far smaller than their initial size
  • the dialogs with listctrls (import files, manage custom filter actions, and so on) now start a little shorter in height
  • fixed some bad sizer flags in dialogs with listctrls
  • rebalanced content update throttling for quicker correction under heavy load
  • moved the new 5 second break in the gallery parser to a position for quicker status updates
  • the gallery parser will report total urls found when it is finished (even if that is zero) and wait a bit to let you read that
  • fixed services->news to hide if you aren't connected to any repositories
  • autocomplete boxes are now a very slight shade of blue, see if you like it
  • fixed 'launch file externally' for linux and os x
  • fixed 'launch directory externally' for linux and os x
  • fixed 'set up server for me' for linux and os x, I think
  • improved some general external-process-launching code
  • cleaned and improved some of the predicate code

• version 142

  • added wildcards to autocomplete results and file queries
  • autocomplete results will match your wildcard _exactly_
  • a new predicate type will appear at the top of wildcard queries; selecting it will search files with that wildcard
  • the new wildcard predicate can be prepended with a minus sign to exclude from results just like normal tag and namespace queries
  • in wildcard predicates, namespace and/or tag can contain wildcard characters
  • added some wildcard help
  • putting in '***' as an autocomplete query is now a pretty bad idea!
  • fixed some logic in how tags are matched against unusual search input in the db
  • sped up and cleaned how tags are matched against search input
  • fixed some namespace logic in how tags are matched against search input
  • below-character-threshold autocomplete queries will now return all applicable namespace results (e.g. putting in '1' will return [ 1, page:1, chapter:1, volume:1 ], not just [ 1 ])
  • added 'open externally' to launch the file in default external program to thumbnail and media viewer menu
  • added a five second delay between gallery-page fetches in the downloader to reduce chance of 429 errors (this was affecting big sankaku searches)
  • added danbooru webm downloading
  • fixed a typo in the thread watcher
  • fixed a bit-rot bug that was stopping the 'like' ratings filter from launching
  • fixed right click menu in custom filter

• version 141

  • combined mappings are no longer calculated and stored
  • recalculate combined mappings obviously removed from database menu
  • combined mappings no longer have to be recalculated on a service deletion; the accompanying 'this could take a long time' warning dialog is gone as well
  • combined mappings autocomplete counts are now calculated on-the-fly
  • combined mappings tag queries are now performed on-the-fly
  • combined mappings namespace queries are now performed on-the-fly
  • combined mappings zero/non-zero tag count queries are now performed on-the-fly
  • combined mappings regular tag count queries are now performed on-the-fly
  • corrected some logic with regular tag count queries, I think
  • autocomplete tag cache dirties and culls itself more efficiently on tag update
  • autocomplete tag cache dirties and culls itself more efficiently on file import/delete
  • removed a couple of non-useful AMP tests that were breaking due to a previous change in connection code
  • improved how popup messages give the gui permission to update; hopefully the gui will lock up less when big jobs with popups are happening
  • improved some misc timing logic
  • improved some repo sync timing logic
  • added simple emergency throttling to the repo sync daemon when the CPU/HDD is getting hammered
  • improved some repo sync text number-grammar and timing
  • added sankaku channel booru, including flash
  • the booru downloading code that discovers image urls is more flexible
  • improved my job pause/cancel logic so paused jobs, when cancelled/completed, will report themselves as no longer paused (this was affecting the downloader page, with paused jobs not dismissing themselves on a subsequent cancel)

• version 140

  • if a repository or subscription popup message has nothing to report, it will dismiss itself
  • fixed handling of text popup display when the object passed was not text
  • delete orphans is now cancellable
  • vacuum, deleted orphans, and upload pending popup messages will dismiss themselves an hour after they are done
  • tightened the subscription final state popup message to just a title and a button
  • removed much of the very expensive autocomplete tag cache maintenance code, which seems not to be worth the effort
  • culled the autocomplete tag cache in prep for new maintenance cycle
  • fixed a resize timing bug that was causing large images to scale in an ugly way when the media viewer was launched in a borderless state
  • 'open selection in new page' will no longer default focus on the sort dropdown; it'll go to the media panel (this was causing scrolling confusion)
  • fixed a non-updating display bug when resizing frames/dialogs with auto-resizing listctrls on linux
  • cleaned up a wall-of-text error when closing the client immediately after deleting a tag service
  • filled a gap in static text image object cleanup
  • cleaned up some thumbnail waterfall/fade code
  • filled several gaps in thumbnail object cleanup

• version 139

  • cleaned up all the old popup message code, and fully integrated the new
  • moved repo sync to the new popup messaging system
  • moved normal errors and db errors to the new popup messaging system
  • improved some error handling code
  • reintroduced message printing
  • improved subscriptions messaging

  • added cancel button to

    • check file integrity
    • export to tag archive

  • added pause and cancel buttons to

    • repository sync
    • subscription sync
    • pending upload
    • regenerate thumbnails
  • improved how jobs' pausability and cancelability are spawned
  • improved and harmonised a lot of pause and cancel and general shutdown-job-interaction logic
  • pausable and cancellable popups can only be dimissed with right click once they are done or cancelled
  • improved some more pause logic in the subscription and repository sync daemons
  • improved popup pause responsivity
  • added comprehensive cancelled/finished log statements for the newly pausable and cancellable operations
  • 'just woke from sleep' calculation will no longer evaluate to true on application start
  • deleted the old broken message log page, which I had forgotten even existed!
  • fixed a bug stopping adding tag repositories in the manage services dialog
  • fixed a bug that was hiding the 'name and credentials' panel in the manage services dialog
  • improved some kinds of thumbnail error reporting
  • sped up client boot by one second

• version 138

  • created new 'maintenance and memory' options page in the manage options dialog, and moved some things over from 'files and memory', which is now 'files and thumbnails'
  • added options for idle, vacuum, and delete orphans periods to the 'maintenance and memory' page in the manage options dialog
  • fixed the incredibly annoying animated scanbar delayed-frame bug, where a click on the scanbar would not draw the new frame until the old frame's expected delay was complete. scanbar scanning is a lot smoother all around, now
  • animations now show their current frame number in the animation scanbar
  • status bar now shows inbox/archived counts on any selection
  • reworked static image zoom code so they resize beautifully, without jaggies, at the cost of a bit of CPU and memory
  • created new flexible message pathway
  • moved most messages to new pathway
  • improved a bunch of message and job_key related code
  • btw: messages will no longer close themselves; their end state is now to report what happened until you dismiss them. if this turns out to be annoying, I'll change it
  • fixed the pending menu not updating its count when files imported and added tags via archive sync
  • improved the logic behind the 'computer just woke from sleep' calculation
  • improved the accuracy of the 'client is currently idle' calculation
  • improved database vacuum so the .db-wal file is flushed afterwards, for both client and server
  • because of better vacuuming, server db backup no longer needs to create a bloated .db-wal backup
  • removed a very common superfluous empty tag upload packet in normal tag uploads

• version 137

  • eliminated a loophole in the tag-pending process that allowed zero-length subtags through during archive sync
  • improved the way tags are cleaned and checked, no matter their source
  • added zero-length subtag (e.g. 'character:') checking and exclusion to the server
  • removed erroneously pending or current tags that have a zero-length subtag
  • added 'copy bitmap' option on still image right-click menu
  • harmonised a whole bunch of clipboard code
  • added a check file integrity db function, with quick filename existence and thorough content hashing modes
  • added clear service info cache debug function to database menu
  • added regenerate combined mappings debug function to database menu
  • added a primary key to the hydrus tag archive db to save some space on duplicate mappings
  • added AddMappings to the hydrus tag archive to make mass mapping-adding a bit more efficient
  • added DeleteTags to the hydrus tag archive
  • added Get/DeleteMappings convenience synonyms for Get/DeleteTags for the hydrus tag archive
  • added an 'export to hydrus tag archive' button in the advanced content update dialog
  • renamed the db tag archive sync methods to better reflect what they actually do
  • massively simplified how cursors are referenced inside the client and server databases
  • cleaned up some db cursor code
  • fixed a critical error in the database test code that was spawning a double db mainloop
  • improved error handling when regenerating thumbnails
  • fixed error handling when waterfalling thumbnails
  • fixed a typo in export tags process
  • improved the way the client flushes its buffered log data to disk
  • added log buffer flushing to the server
  • added a confirm dialog to advanced content update dialog's powerful 'go' button
  • added custom titles to many co-opted yes/no dialog boxes where the default 'Are you sure?' didn't make sense
  • cleaned up some misc code
  • cleaned up some misc bitmap language

• version 136

  • added tag archives
  • added tag archive sync initialisation on all existing local files
  • added tag archive sync maintenance on newly imported files
  • added a new db directory, client_archives
  • added tag archive sync options to tag services in manage services dialog
  • added local tags service to manage services dialog
  • added sha512 to local hash cache
  • added tag archive hash_type guessing
  • added a new dialog for selecting n arbitrary strings
  • got testing to work on all platforms
  • fixed the hydrus server for linux and os x; it now has a stay-alive frame rather than a taskbaricon
  • improved some dialog testing code so it would work on all platforms
  • fixed a deffered problem that was causing the server AMP test to hang on Linux and OS X
  • neatened and harmonised a bit of common file and network streaming code
  • improved some misc manage services dialog code
  • fixed a critical bug that was meaning certain service changes were not being saved to the database, so were being forgotten on restart
  • fixed some select-string code that wasn't taking sets of strings for a weird wx reason

• version 135

  • added a menu option to any tag's right-click menu to open a new search page for that tag
  • added a subscription cache to the client database to speed up subs checking
  • added a filter to the message popup system so those annoying and 99% pointless PyDeadObjectErrors will not display. they will still be written to the log
  • cleaned up a couple of temporary file deletion errors
  • improved some more temp file deletion error handling
  • fixed a bug that I think was stopping file repositories from being deleted
  • improved the manage services db edit log
  • fixed a bad comparison that was causing superfluous edit actions after a manage services dialog ok
  • fixed a new bug related to displaying non text in the popup system
  • added a 'just woke from sleep' check to all daemons, so CPU heavy stuff like repository sync will not initialise if you just woke your computer. the grace period lasts about ten minutes
  • retuned the way the subscrption daemon initialises (it'll now wait two minutes after startup before firing)
  • fixed a typo that was causing fatten service info to fire more often than it should
  • added a yes/no warning to options ok when the thumbnail dimensions have been changed
  • added a popup message when thumbnail dimensions have been changed to report on deletion progress
  • added account testing to my server db testing suite
  • improved the security of the registration_key->access_key transaction; it'll now generate a new access_key with every call

• version 134

  • updated to wx 3.0.1.1
  • fixed a critical media scrolling bug due to the wx update
  • improved some bad media scrolling code, sped things up a bit
  • fixed 'top' and 'bottom' media scrolling events
  • fixed a typo that meant the default fullscreen media browsing shortcuts were ctrl+ appended rather than working on their own
  • overly-verbose errors and other text popups are now cropped to 1KB for gui display, with a notice. the full message will be printed to the log as usual
  • improved how severe boot crashes are reported
  • fixed a bit of text-reporting code that wasn't handling non-text very well
  • improved handling of a weird popup message manager error
  • fixed an occasional overhasty cleanup error in the checkimportfolders daemon
  • added options for default values for the thread watcher's number of times to check and check period
  • fixed the thread watcher complaining about closing when the checking was finished
  • optimised some id generating code to stop spamming the id cache, which I think was overloading after a while and causing weird PyAssertionErrors
  • neatened autocomplete dropdown service storage and menu id generation
  • improved menu id generation for tagbox
  • fixed opening new petition page from view menu
  • added a 'this might take ages' warning yes/no dialog when trying to delete a tag service
  • added a little popup message info to report on progress when deleting a tag service
  • added some server db testing
  • fixed an error when double-clicking a tag in a page without search predicates
  • updated some help links from mediafire to the new github releases page
  • fixed a typo bug in server db's account flushing code

• version 133

  • reworked the add file process to correct file repository file counts
  • made add file process to calculate inbox and local thumbnail count more efficiently
  • fixed a critical bug in server db creation that was stopping file repository service from working
  • added go/exit fullscreen options to fullscreen right click menu as alternative for default shortcut 'f'
  • improved fullscreen position, size, maximised, and borderless state memory
  • removed old 'fullscreen_borderless' option in local options
  • thread checker now supports 8chan urls
  • thread checker now has 'number of checks still to do' spin control so you have more options to fire and forget
  • improved how the thread checker constructs and passes url information around
  • improved some thread checker timing logic which _might_ have been causing problems
  • improved some thread checker error reporting

• version 132

  • merged two complicated serverside account tables into two simpler tables
  • with this action, was able to clean out a lot of rubbish old server account code
  • made it so accounts can only be on one service. existing (admin) accounts that straddled sevices will have new access keys printed in a text file in the base installation directory on server update
  • rewrote the account object to be simpler and easier to maintain
  • swapped the old rubbish 'account_id' identifier in the account object for the much better 'account_key' identifier
  • harmonised some conflicting account-related variable names
  • refined the way the unknown account is stored and identified
  • split serverside account verification and identification into two separate paths, to reduce chance of security problems
  • reworked account identifiers (a general purpose account identifying object that is used in admin-server interactions) and their associated db functions to be more secure and reliable
  • simplified account data use checking
  • simplified and harmonised the way used bytes and used requests are stored and retrieved in the account and account type
  • with guarantee of account_key uniqueness across entire server, I have simplified session code in several places
  • updated help to reflect the new relationship between access keys and account keys
  • added 'copy account key' button to review services, which will now be the thing for users to use if they need an admin to modify their account
  • fixed serverside credential verification for non-instantiated (still have a registration key) access keys
  • added a bit of explaining text to the 'waiting' autocomplete state
  • fixed a typo that caused errors when deleting files from a file repo

• version 131

  • removed tag service precedence and its various expensive and overly complicated effects
  • recalculated combined tag mappings using new simple union based merge
  • cleaned updatemappings code, and I think fixed some tiny count logic bugs
  • refactored eight overly complicated sub-functions of updatemappings down to two simpler ones
  • fixed up a mention of tag service precedence in help
  • deleted a bunch of old client update code
  • added adaptive throttling to tag updates, so they'll speed up and slow down based on current CPU load
  • refined a bunch of the parent and sibling code, given the new changes
  • updated and cobbled back together a host of finicky parent and sibling and regular tag test code, given the new changes
  • fixed a bit of logic in the censorship code that was causing overly broad namespaced matches
  • fixed error when selecting a specific tag service in a normal search
  • deleting a service no longer throws an extraneous error after it is complete
  • fixed danbooru large pngs, which were being resized in a way that broke some of the downloader's url prediction code and caused 404s
  • fixed a bug in the input custom filter action dialog
  • fixed a misc clientserviceidentfier bug in custom filter initialisation
  • fixed setting numerical ratings from a custom filter (I think) and I think fixed many other custom filter operations
  • fixed several text-entry-dialog-launched-from-dialog bugs
  • fixed, I really think for real this time, the serverside account desync bug. it was in session init, where I had not previously looked
  • improved some popup messages' text
  • improved the way some exceptions are reported
  • cleaned some error reporting code
  • silenced the more-annoying-than-useful account sync failure popups
  • fixed and cleaned server start/stop/restart code a bit more
  • reduced regular CPU hit from resize thumbnail daemon and made it more polite in starting

• version 130

  • fixed youtube downloads
  • fixed remote services not showing in OS X manage services dialog
  • updated 'running from source' help page with some new info for Linux
  • cleaned a little more code
  • fixed bug that was stopping new/modified serverside services booting immediately
  • improved the server-client service key translation code on serverside service modification
  • fixed test credentials button in manage services dialog for unsaved services
  • fixed bandwidth tracking errors for unsaved services
  • deleted some old unused server options code
  • improved server services start/stop code a bit more
  • massively rebalanced the autocomplete maintenance routine after a tag service deletion or tag service precedence change
  • made a mess of some of the deleted tag update processing code, I'll fix it next week

• version 129

  • fixed a typo in launching the ratings dialog
  • did some important code cleanup re a common ambigious variable name
  • improved serverside account tracking in a way that will help future user management clientside too
  • improved serverside session code a bit more
  • improved general serverside login
  • improved serverside access key verification
  • aligned server db update code with the client's only-update-up-to-50-versions rule
  • fixed some buggy repository test code
  • fixed some buggy local booru test code

• version 128

  • fixed right click on thumbnail error
  • improved assumed permission defaults on tag dialogs and right click download/upload actions (i.e. tag dialog's entry A/C will no longer disappear when you desync from tag service)
  • removed account desync on network version mismatch, to stop this problem anyway
  • fixed tag parents not showing in autocomplete dropdowns
  • added sqlite3 command line tool to db folder for convenience
  • fixed a typo in manage server's services dialog
  • some misc code cleanup
  • added default advanced tag options pane to manage options dialog
  • added default advanced tag options to manage subscriptions dialog
  • fixed booru namespaces in manage subscriptions dialog to be booru-specific
  • added default advanced tag options to normal download pages

• version 127

  • finished service_identifier rewrite, phew
  • added an exception catch for full program crashes; big errors will now be written to crash.log
  • added a shutdown check to make sure popup message manager closes properly when certain weird errors occur
  • 'files_service_identifiers_cdpp' is now renamed to the vastly less insane 'locations_manager'
  • removed db update code older than 50 versions
  • a new error message will display if trying to update more than 50 versions in one go
  • many other small fixes I didn't keep track of

• version 126

  • restored a dll that I thought was no longer needed, but was actually doing some weirder gif rendering
  • added 'remove' to fullscreen menu
  • harmonised thumbnail and fullscreen right click menus a bit more
  • added pause button to popup messages for repo update and subscription processing
  • moved service_identifier switchover forward
  • moved all service fetching to streamlined and non-laggy manager
  • changed client options to store default tag repository in a better way
  • changed subscriptions to store their advanced tag options in a better way
  • fixed a 'missing service' bug in advanced tag options
  • remade idle calculation into a much better gui-based rather than db-based test
  • fiddled more with maintenance timing, hopefully for the good
  • new screenshots in the help index pages!
  • improved how auto repo and server setup work and report their status
  • the client's UPnP daemon will no longer spam errors if your IGD doesn't support UPnP
  • fixed a bad db call in server's UPnP daemon

• version 125

  • moved client splash screen and client boot to application event loop (i.e. your mouse won't hourglass over it now)
  • splash screen is now draggable
  • added splash screen for client shutdown
  • cleaned up shutdown procedure
  • added detailed A/C timing options
  • created a new text entry dialog that won't allow ok on empty text
  • hence fixed a startup bug when a session was created with no name (and likely others!)
  • further tweaked idle calculation to improve maintenance routine
  • revised a bunch of the maintenance timer logic
  • fixed some bugs in the maintenance timer logic
  • fixed a daemon error reporting typo
  • fixed another way session data usage tracking could split, server-side
  • brushed some more dust off the server-side session code
  • improved some bad image processing nomenclature
  • took first steps in service_identifier replacement rewrite

• version 124

  • fixed some more broken gifs (those with funky transparency, I think)
  • fixed a critical bug in the subscriptions dialog that was causing subscriptions to be deleted
  • reintroduced old locale code, with minor improvement, to fix number grouping (123,456,789) issue
  • added a short pre-activation wait to all daemons to reduce more wake-from-sleep problems like the UPnP issue last week

• version 123

  • rewrote the old gif rendering code to the new system, so gifs with variable framerates now work again!
  • fixed a bug that was stopping the first zoom of an image from showing if it took more than a frame to render
  • webms are now zoomable beyond frame border
  • webms will now respond to keypresses and mouseclicks when cursor is inside their border
  • added service select to import metadata
  • if repo sync is paused, either globally or individually, repo will no longer try to regularly sync their accounts
  • fixed individual repository pause sync
  • I think I fixed a serverside session synchronisation bug that was causing too-small data usage reports
  • improved complicated tag action descriptions in manage tag dialog
  • hastened initialisation of subscriptions daemon
  • hastened initialisation of subscriptions dialog
  • neatened all object declarations
  • fixed a menu bug in OS X
  • added put a little delay in the upnp daemon, hopefully to stop the error popups after waking from sleep
  • updated an old hacky bit of locale code that was breaking certain platforms of wx 2.9.5.0
  • improved animation and other rendering timings on non-windows platforms
  • improved media focus on non-windows platforms

• version 122

  • fixed server upnp daemon to use correct port, I think
  • added client upnp daemon for local booru; it'll update upnp mappings as soon as they are changed
  • added bandwidth tracking for local booru; it'll even magically update before your eyes in review services
  • bandwidth limits are tested, so exceeding data usage will result in 403 errors until the next month
  • moved local booru css to an external file any user can edit
  • added 'open share in new page' button to local booru review services
  • fixed local booru display for video, flash, audio, pdf and miscellaneous
  • fixed local booru render resolution for all media
  • fixed local booru thumbnails for video, flash, audio, pdf and miscellaneous
  • added caching headers to static file requests to reduce browser-hydrus bandwidth
  • fixed local booru page text's newline formatting
  • added local booru unit tests

• version 121

  • first version of local booru

• version 120

  • improved quality of downsized animation rendering
  • sped up downsized animation rendering in this case
  • neatened animation code
  • fixed a mid-animation resize parameter bug
  • I think I fixed an animation scan bug that was sometimes giving the wrong frames
  • added thumbnails for all video formats
  • fixed an off-by-one framecount bug for certain videos, and retroactively fixed counts for existing videos
  • fixed a couple harmless width/height numpy.shape switcharounds
  • cleaned up some file parsing code
  • added a pixiv unit test
  • fixed a bug when servers were returning unusual relative redirect urls (gelbooru)
  • semi-fixed a bug when servers were returning unescaped redirect urls (gelbooru)

• version 119

  • fixed an overzealous error in v118 update code
  • new direct ffmpeg video decoder for webm
  • new system takes advantage of multi-core processors, increasing render speed
  • improved animated frame timings
  • fixed a number of gifs that were causing the "None has no 'shape' attribute" error
  • sped up memory and subprocess cleanup after video playback
  • added some new ffmpeg instructions to the running from source help page
  • added client and server startup and shutdown statements to the log
  • improved some idle time logic, which should improve maintenance timings
  • fixed a critical idle timing bug that was adding 1s to many actions, including file queries!
  • fixed a pubsub testing bug that was preventing daemons from quickly closing after a unit test

• version 118

  • improved animated frame buffer size calculation, saving memory and improving smaller animation performance
  • improved quality of static image thumbnail generation
  • fixed webm mouse event propagation
  • updated and improved perceptual hash code

• version 117

  • fixed the gif rendering error that was causing threading lockup
  • fixed a bug in the animated drawing code that was causing +5ms duration on every frame
  • improved frame duration checking code
  • lowered quality of animated frame resizing to improve effective framerate for previews
  • fixed a problem with pixiv session management
  • fixed a problem with pixiv tag subscriptions
  • neatened an error with pixiv manga download failures
  • corrected reviewservices hyperlinkctrl initialisation
  • improved the wording for the 'port already bound' error message on client startup
  • improved hydrus network session error handling
  • cleaned up some old gui messaging code
  • some general image/video code nomenclature cleanup
  • changed http connection maintenance daemon to be snappier on client shutdown
  • subscription minimum check period is now 1 time unit, not 0!

• version 116

  • gifs now render their frames as needed, rather than building a giant memory-hogging cache of all of them
  • webms now have thumbnails!
  • refactored important hydrus threading code to a separate module
  • added new 'call-to' worker threads for snappier job-thread behaviour
  • fixed a typo that was stopping new repositories being entered in the manage repositories dialog

• version 115

  • moved review and manage services windows to a local/remote dichotomy
  • moved local booru service code generally forward
  • added local booru service to db init/update
  • added booru to review services
  • added booru to manage services
  • neatened some misc client code
  • improved static/animated image rendering code in general
  • added some cv stuff to gif rendering code
  • cleaned up gif rendering code
  • sped up animated gif rendering by about 5-10 times!
  • added some cv stuff to static image rendering code
  • cleaned up static image rendering code
  • sped up static image rendering up to 4 times!
  • image import should generally be faster
  • pulled animated images off the image cache, reducing memory usage
  • added some clever compression that cuts image cache memory usage in half!

• version 114

  • gif rendering seems to be fixed in all cases! hooray!
  • fixed 'pop from empty list' popup error spam in the new cache clearing system
  • fixed weird behaviour on right-clicking 'dismiss all' popups button
  • added a unit test for perceptual hash generation, with an eye to moving the delicate code from cv to cv2
  • updated opencv to 2.4.9
  • updated pyinstaller, so frozen releases may be a bit more stable!
  • moved webm and mkv info parsing over to cv, which allows num_frames

• version 113

  • added mkv+webm support!
  • fixed some system:mime selection stuff
  • fixed a missing db call in the server upnp daemon
  • fixed upnp dialog's edit button
  • cleaned up upnp dialog's post-button refreshing logic
  • fixed getupnpmappings to deal with >10 mappings
  • fixed fullscreen_switch for custom filter
  • fixed a late canvas drawing exception when fullscreen window was closing due to being media-empty
  • rolled back to wx 2.9.5 for windows again. not worth the trouble, right now
  • hydrus caches should now clean up anything older than twenty minutes
  • 'last session' will now be saved every five minutes, rather than only on a successful close

• version 112

  • A/C dropdown is fixed on linux!
  • wrote new UPnP library based on miniupnpc multiplat executables
  • integrated new UPnP library into existing client code
  • upnp management dialog should now work for os x and linux
  • integrated new UPnP library into existing server code
  • added UPnP unit tests
  • fixed a bug when upnp executable path had whitespace
  • renamed 'add, remove or edit services' to 'manage services'
  • fixed some review services frame text display issues for restricted non-repository services
  • optimised review services frame initialisation db requests
  • added a database maintenance routine that should help review services launch a bit faster on average
  • improved manage services layout a little bit
  • removed initialise server button when it is invalid
  • updated 4chan thread url parser to handle new 4chan thread urls
  • redirected 4chan thread downloader to new api address
  • fixed an error reporting problem in repository sync
  • fixed my linux dev environment, so linux executable is back
  • changed wx requirement to 2.9 on linux, due to bugs in linux 3.0.0.0 release
  • fixed a typo in repository deletion
  • added repository deletion to unit tests
  • added num_base and num_step to file path regex dialog
  • fixed popup display of certain exceptions, including file parsing exceptions
  • cleaned up os x and linux build scripts, added static&help sync

• version 111

  • fixed booru searches that include unusual characters like '&'
  • added pause synchronisation to individual repositories in the manage services dialog
  • fixed gui_sessions with predicates (again!)
  • added a unit test for gui_session storage, shouldn't get problems with that again
  • fixed a getcountlesscopy predicates bug
  • first step in multi-platform upnp support is DONE, phew
  • harmonised subscription file import errors with normal hdd import errors. they nshould be a bit clearer
  • corrected how popup text width is enforced
  • added a warning to the reset cache button in the manage subscriptions dialog
  • os x autocomplete dropdown seems to be magically fixed, not sure how or when that happened
  • started HydrusBooru
  • fixed some problems with twisted startup/shutdown in unit testing

• version 110

  • fixed a variable overwriting issue in the subscription daemon error reporting code that was causing error spam
  • fixed more actual and potential instances of this error elsewhere
  • fixed a bug in the import file error system for non-local files
  • fixed url parsing for urls lacking 'http://' or 'https://'
  • fixed hentai foundry image parsing for two different cases
  • fixed client gui-initiated db backup to also backup client_updates folder
  • A/C dropdown now shows current and pending counts separately!
  • fixed display of uncaught database exceptions
  • new version of sqlite for windows, might speed some things up a bit
  • upgraded a bunch of other libraries, look forward to mysterious new bugs!
  • I _think_ some transparency support is improved
  • moved messaging forward
  • dropped processed_mappings experiment
  • removed a bloated mapping index, hopefully speeding a couple things up
  • fixed an erroneous error message in hydrus network session manager

• version 109

  • started processed_mappings table. for now, it mirrors normal mappings table
  • improved manage tags dialog logic
  • fixed only_add issue for parent tags
  • fixed an important tags box surplus tags display bug when focus repository was changed
  • fixed petitioned tag counts in tag boxes, as long as 'all known tags' is not selected
  • fixed occasional zero tag count in tags box
  • added petitioned tags to normal media fetch, not sure why they were missing
  • fixed a random error when trying to refresh a 'open selection in new page' page
  • improved help to reflect new manage tags dialog
  • cleaned up an annoying spam-text-to-console issue when running test.py
  • reworked the way the server clean/create update daemons interface with the database
  • improved how updates are stored and fetched in the server db--no longer needs a db hit to fetch
  • harmonised client db's update naming convention to the server's
  • improved server db to update the same new-ish way client db does it

• version 108

  • added 'database->delete orphan files' for manual firing of this maintenance routine
  • improved redirect location parsing--should fix the booru 'Could not connect to None!' errors
  • fixed 4chan thread downloader's erroneous 'still working' warning when trying to close while paused
  • added multiple additions to manage tag parents dialog
  • added multiple additions to manage tag siblings dialog
  • generic improvements to both dialogs
  • manage tags dialog now shows all the selection's tags combined, not just the tag intersection
  • multiple improvements to manage tags dialog
  • reworked some tagsbox classes to be more flexible
  • improved dialogsetupexport's use of tagsbox

• version 107

  • converted 'namespace blacklists' to more general 'tag censorship'
  • tag censorship now accepts unnamespaced tag bans
  • tag censorship now accepts all namespaced tags bans
  • tag censorship now permits bans for all tag services
  • fixed several small bugs carried over from the namespace blacklists code
  • added a bit about tag censorship to help, under advanced.html
  • fixed youtube downloading
  • file import dialog now ignores dupe paths
  • improved update status text in review services window
  • fixed erroneous capital P on pending menu
  • fixed shortcut input dialog's ok button
  • fixed modify tags dialog's admin modify button
  • added several bits of explanation text to advanced dialogs
  • added more helpful timestamp to news dialog
  • neatened export dialog's layout
  • corrected some double-escaped backslashes in the export filename pattern button menu
  • fixed modify accounts dialog's http request
  • improved grammar in db pubsub code

• version 106

  • download by raw url now sends the correct text to the popup on successful import
  • first step in peer-to-peer repository sync is complete
  • repository updates are now stored as local yaml, in the new client_updates folder
  • repository reset keeps the downloaded updates, only does a reprocess
  • all repository updates should process faster, especially if you have saved sessions
  • a lot of misc gui lag should be reduced, especially if you have saved sessions
  • the way the gui's menubar is updated is massively improved
  • os x and linux menubar should be less spastic
  • several improvements to daemon pubsub workflow

• version 105

  • complete rewrite of message data handling and display
  • hidden gauge messages will now display correctly
  • fixed error-display bugs in the log page
  • massively improved how database errors are generated and displayed
  • improved and streamlined how other errors are generated and displayed
  • improved the way all gui elements are deleted from memory
  • fixed the frequent segfault in os x and linux when closing popup messages
  • fixed the common segfault in os x and linux when refreshing a query
  • fixed the occasional segfault in os x and linux when closing a page
  • fixed pattern shortcut buttons
  • the 'show deleted' checkbox on manage tags dialog makes a return
  • neatened several checkboxes so clicking the label text clicks the checkbox

• version 104

  • first version of export folders is ready
  • cleaned up all ok and cancel dialog behaviour
  • cleaned up some misc dialog button code
  • fixed up some ugly db yaml pubsub stuff
  • fixed a juddering 100% CPU loop in the subscriptions code
  • subs and repo sync will now put up a popup gauge
  • repo sync will now pause mid-update on repository pause
  • services->pause->stuff will poke daemons to restart more reliably
  • removed the service status area of the statusbar
  • fixed 'subscriptions forgetting their tag options' bug
  • added select invert, thanks to @fluffy_cub for contribution
  • fixed a slight logic bug with display of select archive/inbox
  • added better instructions on how to run in OS X from source, thanks to user steenuil for contribution

• version 103

  • fixed a virtual size issue with expanding collapsible panes inside scrolling windows :S
  • fixed images not showing on first zoom switch
  • improved touch and feel of image/flash scanbar
  • subscriptions dialog is now flattened
  • reworked advanced_tag_options to be more efficient and flexible
  • fixed a couple bugs in advanced_tag_options
  • moved fake http unit tests to a fake version of the new networking system (so much better, now!)
  • moved real http unit tests to new networking system (so much better, now!)
  • fixed up a last couple network things
  • I probably fixed some file upload code, as well
  • cleared out old network code
  • fixed a fake pubsub problem in unit tests

  • made a single new db table for consolidating

  • 4chan_pass
  • pixiv_account
  • boorus
  • favourite_custom_filter_actions
  • gui_sessions
  • imageboard_sites
  • imageboards
  • import_folders
  • subscriptions
  • couple small help edits
  • fixed view->new booru download page
  • started export folders

• version 102

  • initial linux release
  • rewrote running_from_source.html
  • altered some service object and db stuff to look after null access keys and temp services a bit better
  • key registration works on new network system
  • all downloaders use new network system
  • A/C timer tweaked a little more
  • did some timer event grammar improvements
  • improved button behaviour in import files dialog
  • backing up a server no longer causes a timeout problem
  • prototype export/import tag metadata is done
  • rewrote the way animated gif frames are navigated and thrown to screen
  • fixed an issue that was causing some dialogs to OK when escape was pressed or the close cross was clicked
  • this above issue included the enter system predicate dialog

• version 101

  • fixed another update problem for clients <v95
  • uploading pending content should be less gui-blocking
  • fixed numpad navigation of page chooser to work for qwerty rather than dvorak as default
  • os x page picker now obeys arrow and numpad keys for navigation
  • os x command key should be treated as ctrl now, for better cross-platform compatibility
  • a couple more cmd/ctrl confusions cleared up
  • os x menubar disappearing after fullscreen problem fixed
  • added stateless http connection manager
  • simplified how http works in hydrus
  • http connections now used more efficiently
  • better cookie parsing
  • better http application shutdown handling
  • some misc http bugs fixed
  • new http error handling should limit occasional spam-errors
  • moved all hydrus http connections to new manager
  • moved many non-hydrus http connections to new manager

• version 100

  • MVC import_controller system created
  • download gauges lag reduced
  • download cancel button desync bug fixed
  • download error code harmonised
  • download async code pushed to threads
  • download job key checking and management improved
  • download queue feed timer will obey page-hidden pause status
  • download success and exception handling harmonised
  • download on-close thread-signalling much improved
  • download pages now take responsibility for download code factories
  • download management panel code harmonised
  • download management panels made much more MVC
  • download error reporting is much improved
  • download fail management is much improved
  • download button response is much faster
  • download progress display is split into the easier to read file/queue/builder trichotomy
  • download display indices are less schizophrenic
  • lots of misc improvements to download code and nomenclature
  • thumbnail page's refresh and select menu items now only appear when appropriate
  • thumbnail page's select menu will only show inbox/archive sub-items when appropriate
  • thumbnail page's menu is less buggy generally
  • db updates are now parcelised into each version, rather than one big job
  • improved db version update notification
  • a problem with updates from ~v70 to >v95 is fixed
  • fixed a bug in popup message dismissal
  • database exception formatting improved
  • database exception display spam reduced
  • database exception redundant traceback removed
  • autocomplete character-search-delay time extended from 150ms to 250ms
  • async wx window destroy code improved in certain places
  • improved non-wx-thread popup error reporting
  • some other bugfix, grammar and nomenclature stuff I can't remember

• version 99

  • added backup database menu option
  • added restore database menu option
  • gone back to wx 2.9.5 for now
  • fullscreen scrolling should be a bit smoother now
  • fixed sessions for 'open selection in new window'
  • new regex shortcut for filename in the 'add tags before importing' dialog
  • fixed dialogs cancel-closing on an unfocused 'enter'
  • finally fixed the animation scanbar vertical sizing on preview screen
  • animation scanbar now shows gif frame rendering progress more smoothly
  • test.exe now included in windows release
  • some frame parents code cleanup
  • a tiny bit of gif code cleanup
  • deleted some old server code
  • cleaned up import pause/cancel code
  • improved hydrus's thread-communication objects
  • cleaned up downloader code and logic and gui display

• version 98

  • update to wxpython 3.0
  • you can now add tags when importing from a zip
  • you can now download arbitrary urls from download->a raw url

  • 'select files to import' dialog has massively improved file parsing, with

  • new gauge location
  • queueable parse jobs
  • paths that stream as they are parsed
  • pause/cancel buttons
  • new 'check credentials' button in manage services dialog
  • new access_key_verification server request
  • fixed a very important bug that affected normal tags when adding parents or siblings
  • fixed a bug where old pending parents would not be remembered when new same-child parents were later pended
  • fixed a similar bug where rescinding parent pends would screw things up
  • pending counts will new correctly update on a sibling or parent pend rescind

  • parent tags no longer (incorrectly) remove when

  • removing 'just this file' child tags in 'add tags before importing' dialog
  • removing child tags in custom filter
  • some youtube videos were not downloading due to a redirect issue. it is improved, possibly fixed completely
  • collapsible panes, like 'advanced import options', will refit dialogs on collapse/expand
  • fixed an bug in the way certain server errors were handled
  • fixed a serverside bug for GET account when using account_id as identifier
  • fixed 'modify petitioner' button in petition pages
  • fixed a bug that was preventing clearing of temp path on client startup
  • cleaned up some credentials code
  • some general code grammar improvement
  • reworded some help
  • reworked account identifiers
  • os x
  • now has an app icon, woop
  • os x page closing logic improved
  • collections checkbox dropdown seems to work, now
  • fixed segfault on popup message close
  • thumbs loading from initial session seems to draw properly, sometimes
  • fullscreen no longer causes segfault whenever you zoom or scroll

• version 97

  • initial os x test build
  • collect typo fixed
  • 'remove all mappings with specific namespace' added to advanced content update dialog

• version 96

  • went over _all_ my help (about 200KB of text!), cutting down waffly paragraphs, rewritting bits that were stiff or unclear, updating screenshots, and adding several new sections people suggested
  • did a complete rewrite of how the client stores services in its database, streamlining everything
  • several bugs exposed by the services rewrite fixed
  • several moments of weirdness exposed by the services rewrite made more sane
  • new version of sqlite (3.8.2)
  • new version of python (2.7.6)
  • cleaned up a debug print statement that was spamming gauge numbers to the log
  • when loaded from a session, hdd import pages will now hide their processing panel
  • fixed namespace sorting to manage numbers again
  • collapsed the many different download gallery page classes to a single one
  • started an important rewrite of how downloaders are going to work in future
  • improved testing framework's pubsub testing support
  • subs sync daemon now restarts correctly after a dialog-driven pause
  • parents now obey siblings!
  • siblings and parents are updated on forget pending
  • got AMP message between persistent and temporary peers working

• version 95

  • pages launched from a session will no longer run their query on startup; they will remember their thumbnails
  • import pages added to gui sessions, although they do not yet remember their progress
  • the way session pages are started is improved
  • advanced content update dialog simplified and rearranged
  • advanced content update db code written for copy operation
  • advanced content update db code written for delete operation (local service only)
  • advanced content update db code written for delete deleted operation ( local service only)
  • made first version of IMMessage objects
  • added network version checking to hydrus's AMP
  • better AMP error catching and reporting added
  • AMP tests moved further forward
  • tightened up AMP network code, fixed last bugs
  • cleaned up a huge pile of 'expiry/expires/expiration' semantic mess
  • same for some mapping_ids/mappings_ids stuff
  • removed predicate knowledge from media display classes, which was just a mess, really
  • corrected a weird param order in sessions
  • general code cleanup
  • several old instances of tuple-list yaml compromise removed
  • fixed a session bug in the testing code because of last week's manager management change
  • fixed a hydrus session bug for accounts with no expiry
  • fixed a very important server-side network version checking bug
  • added client-side network version checking

• version 94

  • reorganised the 'add tags from path' dialogs so the regex buttons made a little more sense
  • when deleting a file, the current focus will pre-defocus
  • 'confirm close client' dialog will now auto-yes after 15 seconds
  • fixed a bug when POST hydrus queries couldn't return any data
  • 'failed to update repository' message made simpler
  • added tuples to pyyaml's safe_loader (which was more of a pain than I expected)
  • manager management improved, with changes reflected across all the code
  • session management prototype is done
  • query pages can be added to sessions
  • default startup session can be set in options
  • sessions are managed with new submenu under file
  • creating registration keys will show them in a new, more helpful frame
  • same for initialising a server
  • and for auto server setup
  • auto server setup has cleaner error reporting
  • new service admin accounts now have different access keys to the server admin account
  • fixed up testing framework to test AMP a bit better
  • IM session manager written
  • server db support for new session manager added

• version 93

  • can now undo/redo all tag operations except rescind petition
  • can now undo/redo file pending, rescind pending and petition
  • improved undo/redo description generation
  • fixed a bug that was showing extraneous undo/redo actions that did nothing
  • review services frame's permissions strings will size a little better
  • new 'perform a service-wide update' button on review services frame
  • started tag-specific dialog for this new button, and began db code that'll execute its stuff
  • a little code reorganisation
  • fixed a bug in the export dialog that was throwing errors when trying to overwrite read-only files
  • moved a number of old dialog error messages to the new verbose popup system
  • when accounts are made stale (as part of client version update), they will no longer lose their privileges
  • fixed the log page to deal with (ignore) gauge messages
  • fixed a db problem in my testing framework
  • AMP IM support moved way forward
  • IM manager first draft done

• version 92

  • encrypted instant messaging framework started
  • encrypted instant messaging test added
  • encrypted instant messaging trust framework started
  • started an AMP framework for encrypted instant messaging
  • tag A/C exact match parameter added
  • tag A/C exact match parameter test added
  • tag A/C logic reorganised and improved
  • tag A/C now gives exact matches for queries shorter than the A/C threshold
  • tag A/C threshold default set to 2
  • fixed an important architectural bug in pubsub
  • generally cleaned up pubsub code
  • 'popup messages sometimes not updating' problem is fixed
  • a bit of initialisation cleanup to make startup behaviour more reliable

• version 91

  • improved how accounts are identified in the server
  • rewrote how serverside session data is managed
  • added monthly data reset to session data
  • fixed session update on account modification
  • fixed a bug in accountidentifier generation from accounts
  • fixed a bug in account-multi-session tracking
  • server session manager unit test added
  • reworked testing to take arguments to only run specific tests
  • changed all file downloads to be more memory efficient, either by gui or subscription
  • made download testing fail better
  • added deviant art unit test
  • fixed deviant art thumb parsing error
  • fixed deviant art fullsize image parsing error
  • undo system reworked in prep for first prototype
  • undo/redo keyboard shortcuts (ctrl-z and -y) infrastructure added and db updated
  • undo works for fullscreen
  • a bunch of fullscreen keypress processing improved
  • undo manager testing added
  • added upnp option to all services
  • added upnp daemon to maintain upnp serverside
  • 'copy all tags' and 'copy all tags with counts' added to tagslists
  • rejiggered a bit of the app startup, including db password check
  • found a better error noise
  • in fullscreen, the mouse cursor will no longer auto-hide when over an animation scanbar

• version 90

  • client db unit tests added

  • 4chan_pass
  • autocomplete_tags
  • booru and boorus
  • downloads
  • favourite_custom_filter_actions
  • imageboard
  • import_folders
  • md5_status
  • media_results
  • namespace_blacklists
  • news
  • pixiv_account
  • improved services
  • sessions
  • shutdown_timestamps
  • fixed modify account
  • fixed modify account again!
  • made account_info respond a lot faster with large tag counts
  • neatened some client db code
  • youtube 'file already deleted' error now reported correctly
  • youtube general error reporting improved
  • fixed multiple-computer-single-account data use reporting
  • moved file parsing from nested dialogs to a single popup message gauge
  • fixed an export folder path-display issue when an install moves location
  • improved how some file downloaders process their files
  • added prototype file download status bars to all download management panels
  • fixed a bunch of tests that were cleaning up inelegantly

• version 89

  • fixed blacklist manager for numtags
  • fixed a sql typo for numtags
  • delete from fullscreen menu fixed
  • fixed a unicode error-handling issue
  • sped up autocomplete tag fetch for large result sets
  • improved the way autocomplete dropdown scrolls
  • vastly improved efficiency of how all listboxes draw
  • the server will no longer start if something is using the server admin port
  • i.e. you can no longer run multiple copies of the same server by accident
  • improved client and server port change (service shutdown and restart)
  • improved testing framework to handle fake db reads better
  • improved testing framework to manage client and server sessions

  • several network request unit tests added

  • access_key
  • account
  • account_info
  • account_types GET
  • account_types POST
  • file_repo file
  • file_repo thumbnail
  • init
  • ip
  • news
  • petition
  • registration_keys
  • services GET
  • services POST
  • session_key
  • stats
  • update GET
  • update POST
  • fixed clientside permissions exception

  • several db unit tests added

  • import_file

  • system_predicates

  • age
  • archive
  • duration
  • everything
  • file_service
  • hash
  • height
  • inbox
  • local
  • mime
  • not_local
  • num_tags
  • num_words
  • ratio
  • similar_to
  • size
  • width
  • limit
  • fixed a bug in system:age?
  • fixed a bug in system:duration<0
  • fixed a bug in system:duration?0
  • improved num_tags logic
  • fixed system:num_tags<1
  • fixed system:num_tags>1
  • fixed a hash filter bug for system:similar_to
  • fixed system:similar_to for unknown hashes
  • fixed system:size Bytes
  • updated linux.html and running_from_source.html in help

• version 88

  • moved server daemon start to a better place
  • cleaned up more server db code
  • fixed getnumpetitions
  • fixed getpetition
  • reworked modifyservices and fixed server request
  • reworked modifyservices dialog to work with new system
  • server service options now managed from modifyservices dialog; old options dialogs deleted
  • auto-repository setup restored and fixed
  • harmonised edit_log actions all across the code
  • added namespace blacklist manager
  • added namespace blacklist dialog

  • integrated namespace blacklist into

  • • search by tag
    • search by tag count
    • creating media results (thumbnails)
    • autocomplete tags read
    • autocomplete tags write
  • fixed a problem with deleted tags not being applied correctly on initial thumbnail display
  • started a server unit testing framework
  • added 'running in linux' info to help

• version 87

  • misc

  • • fixed system:untagged, which was doing numtags>0 by accident

  • basics

  • • moved client local service to twisted
    • moved server to twisted
    • cleaned up a whole lot of unrelated server stuff I haven't touched in a while
    • fixed some misc typos

  • details

  • • built twisted server framework to plug into hydrus
    • changed hydrus authorisation header
    • changed sessions to manage accounts, reducing server db load
    • reworked session cookie to be neater
    • cleaned up a bunch of server code
    • fixed how certain server errors were being printed to the log
    • reworked response_context to a body/path dichotomy to reduce cpu+memory on file requests
    • collapsed a bunch of GET POST requests to the same path
    • reworked server_service_identifiers to be port-independant
    • improved data use logging to support new session management system
    • moved session management to new system
    • moved error management to new system
    • integrated twisted thread into hydrus controller
    • reworked root and favicon
    • reworked local file and thumbnail requests
    • reworked restricted service requests
    • reworked admin service requests
    • reworked repository service requests
    • removed manage options query; it'll be rolled into manage services admin queries
    • updated serverdb to manage with new system, harmonising internal and external requests to one workflow
    • made server-side data use tracking simpler
    • added Content-Type header to most requests
    • improved how key registration, init, and account GET requests are handled client-side
    • harmonised how account_keys are created server-side
    • moved server management from serverdb to servercontroller, with better pubsub restart
    • moved several useful functions to the new serverconstants.py
    • fixed a max_age sessions issue

• version 86

  • timeout on connections improved
  • rewrote the objects behind the dumper
  • the dumper should now select media properly again
  • closed pages now timeout after an hour
  • upgraded to new version of sqlite
  • fixed client port bind detection and error reporting
  • you can set the client's local server's port in client options
  • local server port changes will happen when dialog changes
  • unified the upload and youtube download popup messages into one popup
  • fixed changing thumbnail size (stupid typo!)
  • added 'database->regenerate all thumbnails' to fix certain thumbnail errors
  • add a popup gauge for regen all thumbnails
  • fixed yesno dialogs, which were showing a neutral 'Cancel' button rather than a red 'no'
  • the upnp dialog's buttons now work, have fun with it!

• version 85

  • updated to new version of wxpython
  • several key_down events moved to char_hook
  • rewrote collapsiblepane class to remove weird refit and scroll behaviour
  • a problem with adding service via a registration key is fixed
  • tag upload pending is granulated into separate queries, to reduce server and client lag
  • uploads moved to an asynchronous popup message with a gauge, rather than the modal progressdialog
  • reorganised file upload content_updates to be a bit smoother
  • fixed a redirection bug that wasn't obeying schema changes such as http->https
  • updated e621 url to https schema

• version 84

  • switch fullscreen button added to fullscreen canvases
  • messaging stuff is disabled for now
  • gif scanbars now fill up as the frames render
  • pngs with transparency are no longer drawn with black background after first viewing
  • made a small change to listbooks to correct some gui-weirdness, particularly in local options
  • autocomplete dropdown windows will now hide-and-reposition on parent scroll events
  • autocomplete hide-and-reposition waits 250ms rather than 100ms, making it a little less flickery
  • a graphical bug related to hitting end on a large search is fixed
  • upnp framework started
  • services->manage local upnp started
  • new messagegauge added
  • fixed mp4 import
  • youtube url->formats added
  • youtube format chooser dialog added
  • youtube downloader thread added
  • youtube gauge popup added
  • youtube error handling improved
  • youtube num_bytes_done added
  • youtube unknown total_num_bytes handled
  • youtube cancel button added
  • youtube right click dismiss throws up a yes/no dialog to decide whether to cancel the download
  • started manage upnp dialog
  • closed page undo added
  • closed pages will be paused and quiet
  • pause import folders option added
  • fixed an invalid index drawing bug after removing certain media
  • thumbnail waterfall improved
  • tags are limited to 1024 characters

• version 83

  • sort by longest fixed for files with no duration
  • reverse sort by rating fixed for files with no rating
  • sort by largest, newest fixed for files with unknown size or timestamp
  • sort by unknown/absent values sorts more accurately
  • search by num_tags and min_num_tags fixed
  • remove media thumbnail update bug fixed
  • currently viewable indices screwed up after a remove media bug fixed
  • remove all media entire black screen bug fixed
  • delete media redraw optimised
  • several message print unicode/raw byte errors fixed
  • scrollbar position calculation improved, so 'black rectangle on scrolled thumbnail canvas when num_cols changed after resize' bug should be fixed
  • 'tiny black lines on small slider drag' bug fixed
  • moved a bunch of right_up events to right_down, see what you think
  • popup message cleanup is better on shutdown
  • pubsub typeerrors handled more gracefully
  • improved how options init and update works
  • client will now remember the restored size, restored position, maximised state, and screen the gui was last left at
  • client will remember all the same details for fullscreen, separately
  • these sizes _should_ rescue from offscreen if you disconnect/reposition a non-primary display
  • shift+home/end now scrolls and selects, rather than only buggily scrolling
  • I fixed a content_update bug re petitions
  • 'show in new page' pages will no longer show search subbox
  • added hentai foundry tests
  • fixed hentai foundry title and creator tag parsing in certain cases
  • fixed some related html unicode issues
  • initial value on tag siblings dialog fixed
  • tag siblings and parent dialogs now start sorted by the right column (parent/older sibling)
  • setting a default collect will no longer cause a nasty crash!
  • setting an orphaned default collect will no longer cause problems
  • deselect thumbnails has improved focussed media management
  • can now select inbox or archive from right click menu

• version 82

  • a bug where slow search results would sometimes appear after search predicates were removed has been fixed
  • a lot of autocomplete gui- and db-blocking reorganisation
  • searches are now entirely asynchronous to gui thread
  • searches are split into two granular phases, and are cancellable during processing
  • simplified system predicate storage
  • consolidated all system predicate filtering to initial db search
  • huge improvements to how the thumbnail canvas is sized and extended and drawn to
  • numerous fixes and improvements to how thumbnails are drawn to screen
  • reworked how collect works
  • and sort
  • the raw cpu time behind sorting in muuuuuuuch faster
  • clarified my listening media class
  • added another optimisation to canvas resizing
  • big improvement to canvas redrawing and refreshing
  • important fix in how client figured out what to draw when clicking in whitespace
  • thumbnail fade is much smoother
  • thumbnails should now generally draw a little smoother
  • selectall and selectnone's thumbnail fade is less cpu intensive
  • cleared up a number of collect related selection and index bugs
  • fixed a media identifier issue
  • all import-related append and drawing is muuuuch faster
  • flicker bug on small appends fixed
  • fixed a page up/down drawing bug on the main thumbnail canvas view
  • fixed an unknown-timestamp thumbnail right click issue
  • made my internal media data storage system a lot simpler
  • new sortedlist class to make some media stuff easier to manage

  • reworked how new media is added to a page of thumbs (usually import-append, but the new system supports collect- and sort-sympathetic insertion)

  • -calculating combined tags on the left can be up to ten times faster
  • -integrating new files into the sort is much quicker
  • -integrating new files into the collect is much quicker
  • -improved how new thumbnails are decided to be drawn
  • trying to archive or inbox more than one file, you'll get a yes/no dialog to confirm
  • started db testing framework
  • db tracebacks improved
  • slimmed down content_update data-side processing
  • a couple small code fixes
  • archive and inbox no longer remove from the search if the opposite system predicate is set
  • system:rating fixed
  • fixed a bug where you could middle-click-download files that had no known source
  • couple of bugs with system:ratio fixed
  • system:ratio string representation is 16:9, rather than 1.777779
  • gui thread is more intelligent in telling non-gui threads that it is busy

• version 81

  • mp4 added
  • mp4 mime search added
  • wma added
  • wma mime search added
  • wmv added
  • wmv mime search added
  • changed to video and audio icons, rather than one for each mime
  • popup messages now dismiss on RIGHT_UP rather than RIGHT_DOWN
  • moved many more messagebox errors to the new popup error system
  • cleaned up clientgui's superfluous error handling
  • rewrote error redirection to new popup system
  • more error overhaul
  • print statements overhaul
  • better db updated statement, through popup system
  • traceback show/hide button added to new popup error box
  • copy button added to new popup error box
  • added a 'x more messages/dismiss all' popup when there are more than ten messages to display
  • review services refresh account button will disable until the server replies
  • file system predicate dialogs have better initial focus
  • streamlined server file import and improved respective data usage calc
  • made internal options storage a lot more efficient
  • cleaned up a little cache code
  • export to zip improved to new file handling system
  • upload to repo improved to new file handling system
  • some db write priority stuff switched around and renamed
  • streamlined a little client db workflow
  • neatened client db exception handling
  • db updated message is now a popup
  • removed some orphaned methods in client db
  • client file and thumbnail server requests made a bit faster and neater
  • removed db loop from export and dump file fetch
  • neatened some client and server file 404 exception paths
  • further cleaned up the way file paths are calculated and fetched
  • further cleaned db in relation to files, and removed the old mime cache, for both server and client
  • removed orphaned db export infrastructure
  • add_thumbnails in client made simpler
  • improved http response parsing
  • refactored a bunch of http response processing
  • new exception display system added for caught exceptions
  • fixed a shutdown hang on certain delete orphans failure

• version 80

  • manage tags and ratings dialogs have their initial focus corrected
  • custom filter now obeys tag parents for tag actions
  • fixed an annoying as hell multiple-page shared-thumbnail visual-selection-status bug
  • added e621 test
  • fixed e621 tag parsing, including new species namespace
  • might have fixed a missing thumbnail error
  • fixed system:hash
  • fixed some repo file downloading stuff
  • fixed some general db fetching file info stuff
  • made x files imported from y message a little slimmer
  • fixed minsize for the main gui frame
  • added better initialsize for the fullscreen frame
  • putting a thread url into the thread_id part of the dumper will auto-convert to the thread_id
  • added dumper's multipart form data generation test
  • added generic multipart form data generation test
  • fixed dumper unicode issue
  • error reporting in dumper is a little better
  • streamlined how mime is calced
  • massively improved how mp3, flac and ogg are parsed and validated
  • massively improved the internal import files workflow to use less memory and hdd reads and writes, particularly for video
  • similarly improved server-side import workflow
  • the way temp dir works is improved
  • sped up server file/thumbnail fetch
  • reworked how thumbnails are fetched in a couple places
  • started rework of how options is stored
  • wrote mp4 properties parser, but no mp4 just yet

• version 79

  • popup messages will now report whenever a subscription or import folder successfully imports some files, with a button to show them in a new search
  • popup messages now wrap
  • slightly better error popup
  • many more errors reported through error popup
  • old logging system switched over to new messaging system
  • popup message manager will only show ten messages at once, now
  • selectfromlistofstrings now supports enter key to select
  • completely reworked how dialog cancel works; absolutely all dialogs should now close with escape key
  • reworked how a bunch of dialogs do ok
  • a huge amount of dialog refactoring
  • made ok button initial focus of all dialogs
  • couple of bugs in service options dialog fixed
  • rejiggered some button names and focus behaviour a bit more
  • some classname refactoring
  • muchly improved string->unicode handling
  • improved timestamp generation
  • fixed a clientserviceidentifier->text bug
  • reworked how namespace cache in tagsmanager object is calculated
  • improved instantiation of noneablespinctrls
  • fixed a bug in the thumbnails download daemon
  • changed the way daemons wait for the db, much to the better
  • moved daemons out of the db object
  • rejiggered writedaemon synchrony so exceptions work
  • fiddled around with some help links
  • default, non-maximised size of client is a little more comfortable
  • custom filter now has a popup that'll let you change the custom actions mid-filter
  • server now uses new synchronous logging system
  • fixed an options save bug for server
  • updated server diagram in help
  • added test for dialog selectfromlistofstrings
  • added test for dialogyesno
  • made a framework for testing that requires network stuff
  • made a newgrounds test
  • fixed newgrounds swf parsing
  • made a framework for testing that requires file reads and writes
  • fixed a graceful-exception bug in mime parsing
  • added test for synchronous import_folders
  • added test for delete import_folders
  • wrote a test for importfolders daemon
  • import folders no longer delete or reattempt failed imports; they'll just ignore them
  • import folders are deleted on update, since old objects are obselete
  • import folders won't try to do zips any more; they'll just ignore them
  • rejiggered how import folders does its path parsing to remove mime calc cpu usage

• version 78

  • expanded parents testing with a namespace example
  • made getgstvcp in tags manager only return numbers for vcp
  • refactored ManageDialogs to their own file
  • refactored Exceptions to their own file
  • improved my testing framework so it can do wx gui elements
  • added test for dialogchoosenewservicemethod, fixed a typo
  • added test for dialogfinishfiltering
  • added test for dialogfinishratingfiltering
  • those two finish filter dialogs are a little simpler, now
  • set up a system so dialogs are a bit simpler, in terms of button event processing
  • added test for dialogfirststart
  • added 'add parent to tag' on tag right click menu
  • added 'add sibling to tag' on tag right click menu
  • made manage parents/siblings dialog focus selection more intelligent
  • 'audio - any' mime search fixed
  • reworked audio embed container significantly
  • added embed button
  • all noisy mimes are now protected by embed button
  • you can now rescind pending file uploads or petitions from thumbnail right click menu
  • fixed a redundant bit of code in process content updates
  • in modify account dialog, set account expiry to 'does not expire' fixed
  • logging now happens a little more synchronously for client
  • popup messages prototype done
  • popup message for normal text
  • popup message for some errors
  • fixed a small redundancy bug in data cache
  • fixed a localisation bug when converting numbers (1234) to formatted version (1,234, 1.234, 1 234)
  • improved the code behind 'open selection in new page' in two ways

• version 77

  • tag parents manager unit tests done
  • completely rewrote parents pair generation
  • rewrote parents pair retrieval to be much faster
  • parents manager is much more resistant to loops
  • in two ways!
  • loop detection fixed in manage parents dialog, as well
  • fixed how deleted parents are applied to combined service identifier
  • fixed how deleted parents are applied to specific service identifier
  • tag manager unit tests done, and everything was working ok!
  • shuffled objects around a bit, unified some tag code
  • tag manager merge unit tests done, and everything was working ok!
  • tag manager merge code simplified and result rewritten to a new, simpler class
  • small bug in how preview canvas fetches tags fixed
  • finished off import folders daemon
  • added import folders gui
  • added flac support
  • added ogg support
  • added a secondary layer of mime detection for greater accuracy
  • added a tertiary layer of mime detection for greater mp3 accuracy!
  • fixed an offset bug in mime detection
  • simplified and unified client/server file info parsing
  • import error handling improved
  • neatened how special thumbs are stored, refreshed, and retrieved
  • neatened how the bmp->png conversion happens
  • improved hash->mime caching for both client and server, to speed up http retrieval time for the new filetypes
  • better mime fetch in dumper, too
  • improved debugging messages for a bunch of my custom objects

• version 76

  • made unit testing framework
  • tag siblings object now tested properly
  • hydrusdownloading functions now tested properly
  • added simple mp3 support to client, with external launch or embed
  • added search by mp3 mime
  • added mp3 to server
  • higher precedence deleted tag siblings now overrule lower existing pairs
  • tag sibling closed loop pairs ( a, a ) are skipped
  • neatened predicate sibling matching in two ways
  • improved tag autocomplete matching logic
  • skip logic improved for like ratings filter
  • skip logic improved and button added for numerical ratings filter
  • did a little work on reducing memory usage
  • reordered data cache's memory purge, which will make huge images load better
  • fixed a zoom bug with pdf buttons
  • added some select all, select none and refresh to thumbnail right click menus
  • reorganised menus a bit anyway
  • fullscreen tags on the left are now sibling-collapsed

• version 75

  • fullscreenpopup window added
  • fullscreenpopup window can be dragged about
  • fullscreenpopupfilterinbox
  • fullscreenpopupfilterlike
  • fullscreenpopupfilternumerical
  • accuracy slider moved to popup
  • compare same image until done added to popup
  • keep on left/random/right added to popup
  • 'don't ratings filter this' added to popup
  • that annoying as hell thumbnail selection drawing state bug is fixed
  • display of flash with only one frame fixed
  • downloaded tags now work again, sorry for the disruption!
  • deleted tags will now always show in manage tags dialog
  • fixed the various problems that were stopping DELETED->PENDING->CURRENT working for tags
  • fixed the children can only have one parent bug
  • fixed a deleted tag parents bug
  • a little counting logic improved in the special deleted_pending case
  • fixed a typo re uploading file to a repo
  • main guis statusbar now has a little 'db locked' indicator

• version 74

  • flash scanbar added
  • made new mediacontainer class for media canvas
  • extracted scanbar to separate class
  • made animation frame tracking and control more sensible overall
  • previous_frame/next_frame shortcut now works for flash as well
  • fixed age predicate for > hours
  • fixed an important 'all known tags' A/C count cache bug when multiple services have the same tags for a file
  • if you have resolve_petitions permission for a tag service, tag siblings and parents petitions need no reason
  • slight change in how un-namespaced totals are calculated in A/C read
  • collection now obeys siblings
  • thumbnail upper text now obeys siblings
  • fullscreen text now obeys siblings
  • bit of siblings code cleanup
  • a little changeup to how showselectioninnewpage collects its media
  • /asp/, /gd/, /lgbt/, /vr/ and /wsg/ added to default dumper support
  • trying out making the server non-daemon, to see how that manages

• version 73

  • every file operation from fullscreen was broke due to a single-character typo that got propagated
  • fixed local ratings, which were broken five different ways
  • ditched the deleted_pending concept, which was actually making things more complicated
  • possible fix to review services opening on bottom level
  • dumper no longer says it is still dumping on page close if it is done dumping
  • you can now scroll the dumper's upper comment box
  • moved site links over to github
  • fixed a typo re downloading a file that is already in the db and adding tags at the same time
  • local tags delete fixed
  • tag parents now apply globally, like siblings
  • thumbnail resizer daemon is less verbose and more helpful on IOErrors
  • fixed namespace colours in manage tags dialog
  • open selection in a new page now works for single-file selections
  • upload tags progress bar misalign fixed
  • namespaced counts no longer total up in A/C writes
  • manage siblings dialog now catches collisions when you put in the old, not the pair, and asks you what you want to do
  • age predicate now supports hours
  • while search is not synchronised, A/C will now query db when it might have queried the current media
  • think I fixed a sometimes-not-initially-showing-first-frame-of-gifs problem
  • downloaders now run parsed tags through siblings and parents managers
  • maaaybe fixed a cache counts error
  • wrote a little more sibling and parent help, and updated schema

• version 72

  • remote tag siblings!
  • remote tag parents!
  • simplified update nomenclature
  • streamlined client-to-server update objects and processing
  • streamlined server-to-client update objects and processing
  • did a bunch of work on content_updates
  • did some work on service_updates, too
  • improved client-side petition handling
  • a bunch of changes to how uploadpending works
  • the way pending downloads are stored is improved
  • mappings tables are collapsed server-side
  • file tables are collapsed server-side
  • moved tag siblings and parents to content_updates
  • moved file upload/download to content_updates
  • reworked tag content_updates to be more streamlined
  • addfile/tagrepositoryupdate is made entirely content_updates, now
  • simplification in the way deleted content is stored
  • repo update processing string is muchly improved
  • repo update processing is smoother!
  • a slight change to default behind-the-scenes searches
  • improved how the server creates updates on service init
  • improvements to service_identifiers_to_statuses_to_tags sub-init and related parts of tags_manager
  • some misc nomenclature improvements
  • fixed a denypetition bug for file repositories
  • fixed optimised petition handling in server
  • fixed posting tags in dumper
  • fixed a problem with getunknownaccount
  • server now analyses on vacuum, like client
  • fixed a lolbug in thumbnail resizer daemon that was continually resizing thumbs
  • fixed a bug in cache counts, which should fix the 'pending menu disappeared' problem
  • corrected num_archive count in file repos
  • downloading files is simpler and faster; download menu is simpler
  • fixed server initialisation
  • updated to new sqlite version

• version 71

  • collapsed the four mappings tables into two tables
  • merged the two active_mappings tables into the mappings table
  • made a great number of changes to how mappings and active_mappings are stored and processed throughout
  • 'active' and 'null' nomenclature is now 'combined'; null service_ids are now just ints
  • improved deletepending so it isn't so tough on the a/c cache
  • tags regex dialog entries 'for all files' and 'just for this file' was all broke
  • A/C read now bumps the exact match of the entry to the top of the list, if its count is non-zero
  • fixed A/C for weird-character queries, like '['
  • the dumper now makes success/error noises as appropriate
  • you can turn these noises off in the new sound tab in file->options
  • screwed around with garbage collection while checking mimetypes during pre-import
  • adding a tag parent will spam-add the actual parent tags to every child instance for the appropriate service
  • updated db diagrams
  • revised my sibling chain collapsing algorithm
  • locked db on init dialog message is improved a little
  • system:limit added to combined file service searches
  • num pending menu counts now split into with (pending/petitioned)
  • corrected a server db index oversight
  • newgrounds title tag fixed

• version 70

  • tag parents db stuff
  • tag parents manager
  • tag parents predicate and matches indented expansion

  • and drawing it!

  • tag parents top result reordering
  • tag parents top result insertion
  • tag parents actually doing what they do
  • polished off tag parent help
  • made a full-search table to speed up tag A/C requests
  • added tag full-text search
  • reworked how tag and file services add and reset, to reduce A/C time
  • reworked recalc active mappings to be more beautiful, if not faster
  • fade animation timer improved in several ways
  • thumbnail fetch made much smoother
  • newgrounds artist downloading for games and movies added
  • newgrounds subs added
  • download panel input now highlights on init
  • 4chan filename tag added
  • the setfocus on filter close is neater
  • age phrase is now 'imported [time] ago'
  • age is now shown in fullscreen
  • can now copy file from fullscreen, from menu or shortcut
  • 4chan pass authentication improved
  • fixed a couple tag service precedence sync bugs

• version 69

  • first dialog for tag siblings
  • tag siblings db table
  • tag siblings manager in controller
  • tag siblings display in tagsboxcpp
  • tag siblings display in tagsboxflat
  • tag siblings display in tagsboxmanage
  • tag siblings display in tagsboxactiveonly, via A/C dropdowns, for both read and write
  • top result sibling switcheroo in A/C write
  • A/C db fetch now does siblings too
  • db tag search does siblings
  • siblings help, with some nice mini-charts
  • I fixed collections, which were typo-broke; I'll make sure it doesn't happen again
  • more granular subs error handling, meaning individual file failures won't crash an entire sub
  • individual subs can now be paused
  • e621 fixed for real this time
  • A/C improvement that slows tag updates a little but should stop A/C lag after an update
  • as a result, trying out dropping the CPU-intensive fatten_ac_cache maintenance call
  • A/C read will now update system preds every time you click on it, so inbox/archive counts will stay accurate
  • A/C "all known files + tags" will no longer show the mega-laggy total file count
  • a list -> tuple convenience fix in sanelistctrl
  • if you don't have any pixiv credentials set up, you will now no longer get the option to start downloading pixiv stuff
  • fixed a tiny typo in the thumbnail resizer that made it wait far more politely than was intended
  • slight change to ratio system pred that fixes some lockups, sometimes, I think
  • fullscreen flash and video will get a pixel of whitespace on the right

• version 68

  • fullscreen view now takes addmediaresult
  • export to zip!
  • import from zip, with tag regex stuff too!
  • export to encrypted zip!
  • import from encrypted zip!
  • import encryption help in advanced.html
  • import process code processes mimes less jankily
  • delete after successful import checkbox added
  • import regex tags dialog now has # namespace
  • import regex tags dialog collects its info a bit more intelligently
  • new confirm exit client option
  • subs will save their progress every 20 files downloaded, so pause/restart is less punishing
  • simplified thumbnail cache retrieval
  • some thumbnail waterfall and animation tuning, let me know what you think
  • e621 fixed
  • danbooru fixed
  • reworked booru gallery page num calc, which was completely screwed
  • DA fixed for most cases
  • booru thumb parsing made a _little_ more permissive
  • server now uses the same new file storage system as the client
  • download cancel buttons have a little better feel
  • new thumbnail prefetch thread makes thumbs load a load faster
  • fixed a little rubbish naming semantics in ClientGUI
  • reorganised and streamlined much of the encryption code
  • updated a bunch of help screenshots

• version 67

  • subscription db access improved
  • subscription 'delete subs seemingly at random' bug fixed
  • some nice subs help
  • subs and repos, if changed during a pause, will restart with new changes
  • subs and repos will automatically pause while their respective dialogs are open
  • new namespace | regex listctrl in regex dialog, instead of old sctvcp rubbish
  • new /aa/aa...0 file storage system for client
  • fixed deleteorphans mime issue
  • export and copy files now export writeable files, not read-only
  • rejiggered daemon db access, improving maintenance reliablity
  • dumper and 'show in new page' pages now process content updates correctly

• version 66

  • subscriptions done! works for all normal download types
  • pause repo sync, pause subs sync
  • all download sites moved to the new system
  • lots of small changes to how download code works
  • pixiv tags is renamed to pixiv tag, since it only does one!
  • fixed DA
  • fixed DA again!
  • fixed DA tag parsing for all the diff types of username re http://help.deviantart.com/106/
  • fixed pixiv, somewhat
  • fixed giphy
  • fixed boorus for gallery_advance_num > 1
  • boorus reset in db to default
  • fixed copy files
  • fixed getmime
  • a silly parent assignment for A/C meant they were not closing with pages
  • made the cpu burn for the thumbnail resizer a bit more polite on hdd
  • repo sync daemon and subs sync daemon combined
  • some timing adjustments in sync daemon
  • subscription_type constant -> site_download_type
  • fixed temp folder being cleared on startup (a read-only issue)
  • advancedoptions classes now support setinfo

• version 65

  • added subscriptions dialog
  • added prototype subscription daemon, but not yet activated it
  • added downloader classes
  • began reorganisation of nearly all download code
  • moved parsing around
  • moved advancedhttpconnection around
  • moved serviceupdate around
  • moved some content_update stuff around
  • moved some constants around
  • files in client_files now have extensions
  • files in client_files are now read only
  • pdf launch is simplified as a result of this new ext stuff
  • fixed find similar images; it was just a typo
  • also fixed initial predicates string display
  • rejiggered the thumbnail resizer again; due to my stupidity, it was causing lag
  • thumbnail resizer burns a little more cpu when there are >10,000 or >100,000 thumbnails to render
  • it also now does thumbs in random order, for a couple of good reasons
  • upgraded sqlite, hopefully some queries will run faster?
  • upgraded to python 2.7.4. please report any weird errors

• version 64

  • got rid of system:not_uploaded_to - now system:file service, which is a lot more powerful
  • some display changes to system predicates to make them a bit more human-readable
  • added export dialog
  • made a couple small changes to help about the export dialog
  • collect by rating
  • sort by rating now works for collections, using an estimate (only accurate when also collecting by that rating)
  • built a custom radiobox class and added it to ratings dialog so it has radio buttons again
  • new ratings filter for local_ratings_like services. works just like normal inbox filter
  • added a little info in help about this new filter
  • new session manager for pixiv and hentai foundry, reducing number of session inits the client has to do
  • new init welcoming window for first boot
  • new politer error message for when db is locked on boot
  • shift+arrow keys now pan in fullscreen mode. you can change them in options as normal, if you like
  • resize thumbnails daemon is now part of maintenance thread stuff. hopefully less blocking now
  • inbox pages will remove media when you archive stuff, again. this is just a bit I forgot to update from last week's predicate overhaul
  • fixed and changed the way hitting the change tag service button works
  • I fixed previews and fullscreen cache estimate in local options too!
  • fixed giphy so it works with their new api
  • rejiggered the update notification order to be a bit more accurate
  • updated auto-setup code and help to use new no-ip domain rather than old raw ip
  • v64 will also update the old ip to the new no-ip domain automatically
  • bit of variable renaming
  • cleaned up a couple menu memory leaks
  • updated a little of future.html

• version 63

  • added include/exclude namespace predicate
  • added intelligent delay to A/C to smooth out db requests
  • rejiggered the A/C code so that spamming/pasting several characters and then deleting some will re-search for matches as appropriate
  • cut size down to just B, KB, MB, GB
  • maybe I fixed system:ratio in certain cases!
  • some general predicates display syntax changes
  • (archive/inbox) and (local/not local) mutual exclusivity with new system
  • system:num_words
  • right click -> open selection in new page
  • upgraded from PIL to Pillow, to zero noticeable effect lol
  • had another look at dodgy animated gif transparency; couldn't figure it out
  • CMYK jpegs images now supported! interlaced pngs simply not supported with PIL/Pillow yet
  • numpad delete added as a valid shortcut key
  • removed an old buggy deleteorphans line
  • new thumbnail resize daemon to fill out the thumbnails directory in the background
  • collections with only one file will now be converted to singletons
  • screwed around with locale settings. not sure if this will break things!
  • manage tags and ratings now in fullscreen rmb menu
  • on downloads, ( url -> hash ) pairs are stored in db, even if the file was already deleted or already in db
  • a few additions to help
  • reworked the spinctrls system preds to not have 100 as default max
  • changed a bunch more spinctrls, including memory sizes, in local options
  • and a statictext that wasn't showing number of thumbs for cache size
  • bugfix in ratings filter

• version 62

  • remade a/c dropdown to a better, less buggy class
  • reworked a little of the a/c dropdown focus logic
  • reworked a little of the a/c dropdown show/hide logic
  • a 'sometimes hangs on "gui" on startup after crash' bug is fixed
  • can copy from listboxes with new right click menu
  • dumper page will protest if you try to close it while it is still dumping
  • import page won't protest if you try to close it while paused
  • num_words will show on pdf right click
  • fixed the network version mismatch exception, exactly one week late lol
  • couple of tiny menu memory leaks removed
  • couple of misc gui tidies
  • custom filter favourites save and delete buttons will grey out as appropriate
  • bit of general code cleanup
  • we nested classes now
  • moved contentupdate class from cc to hc, in prep for repo update rewrite and contentupdate gen

• version 61

  • session db tables
  • session manager objects
  • session net code done
  • session failure recovery
  • registration key table
  • registration key requests
  • some registration help added
  • harmonisation of many http request names
  • extracted uploadpending code from db transaction, making it a lot more polite and architecturally correct
  • pixiv now parses the ascii creator name as well as the japanese counterpart. No option to choose between them yet!
  • improved processcontentupdate to support more actions, making db a little more serial
  • home and end now scroll to focussed media
  • some http timeout tweaks, to stop the 'can't connect to server' problems on complicated queries
  • a typo in modify account types was fixed
  • manage tags dialog shouldn't fritz out quite so much when a service doesn't have post_data permisisons

• version 60

  • pixiv account management dialog
  • pixiv gallery parse
  • pixiv page tag parse, including creator and title
  • pixiv user download
  • pixiv tag download
  • collapsed the page chooser a little to fit pixiv in comfortably
  • pdf support!
  • pdf launching via preview/fullscreen button
  • pdf header parsing with num_words rough estimate
  • system:mime=application/pdf
  • system:mime=application
  • custom filter favourites done! with default, previous, save, save as and delete
  • some small changes to temp folder logic
  • vastly improved scroll-thumbnail-prefetch logic. important bits all rewritten and made sane. data and gui rows line up better, with fewer bugs
  • vastly improved thumbnail fade. now attempts to render at 60fps, and calculates alpha much more efficiently
  • thumbnails won't fade if they are off-screen; they'll just draw in one frame
  • fixed thumbnail last row drawing problem (again!)
  • due to daemon-db-spam, db maintenance check was only firing on pc wake from sleep, now it fires properly every twenty idle mins
  • maintenance timer won't fire directly after waking from a sleep
  • deleteorphans copes with alien files a bit more gracefully
  • A/C appearing on top is mostly fixed
  • some general A/C weirdness is fixed/made better
  • reworked some of the thumbnail selection and scrolling code to line up better
  • thumbnail window will stop spamming 'scroll to focus' every time something small changes. it should only do it on fullscreen close and non-ctrl or -shift mouse/key events now
  • some general focus code was looked at
  • ratings filter now supports the fullscreen_switch shortcut
  • just a couple changes to help to include the new 'f' fullscreen_switch shortcut
  • 10s timeout on http queries
  • http redirect code is improved
  • http cookie handling code is improved
  • made hf session establishment a bit cleverer

• version 59

  • new vastly improved 'collect by' dropdown
  • fullscreen 'f' shortcut
  • fullscreen default option
  • added tumblr prototype parser, with tags
  • server init and repository init typos are fixed
  • a misaligned sizer was fixed
  • a bunch of spinctrls no longer init at <= 100
  • the problem with the last thumbnail rows sometimes not drawing properly should be fixed
  • A/C lag calculates a little more intelligently
  • manage ratings for collections now works

• version 58

  • new custom staticbox sizer/panel put in about fifty places
  • big layout changes in management panels
  • I made a new, better set of sizer flags for nested sizers
  • testing out several new borders
  • several other small layout changes, lining things up
  • a few colour changes with the new classes
  • advanced options class is better, more streamlined
  • advanced options class is less flickery, I think
  • A/C spazzes out less when dragged about
  • A/C queries db less annoyingly, especially when you type fast
  • the thumbnail display draws its whitespace a little better, filling in the right-hand and bottom-end gaps
  • importing thumbnails now add to the last row one thumb at a time (rather than at one row at a time) once the scrollbar appears
  • shift + f7 now sends things back to inbox, like a reverse archive.
  • this works from thumbnail menu
  • and in fullscreen
  • new menu item 'remove these' for thumbnails, anywhere, including dumper!
  • 'tags just for this file' now works a looooot better! it unions all selected tags, and detects deselections better
  • giphy downloader added
  • giphy tags added, thanks to @fluffy_cub
  • copying file now _also_ copies paths to clipboard; depending on where you paste, you'll get the files or the text appropriately
  • new shortcut (default ctrl+m) for 'focus media'
  • focus search now works for manage tags and tags regex dialogs
  • think I worked out the last of the up/down arrow cursor shortcut problems
  • a ton more code cleaning
  • hentai foundry artist not found 404 error detected
  • in regex dialog, tags are cleaned before display on screen
  • 4chan ban is now recognised, and thread dumps are hence suspended
  • when ordering tags by incidence, disputes are now resolved by lexicographic
  • fullscreen init is a little faster for the first image
  • select imageboard dialog now toggles tree with double click
  • select imageboard dialog is also taller

• version 57

  • added quick vs accurate slider for numerical ratings filter
  • new slider will remember where you left it last
  • ratings filter internal comparison logic is more sane
  • 'already rated' will now show the specific rating beside it
  • ratings filter (hopefully!) chooses better comparisons, in about four different and new ways
  • you can now ratings filter just one file
  • ratings filter code is neatened a whole lot, especially on my horrible variable names
  • I may have fixed an equality ratings filter bug
  • internal rating data -> pretty text for display is more flexible and accurate
  • going back several steps on ratings filter will no longer forget to re-rate already rated files
  • delete from dumper no longer screws with the indices
  • dumper is more stable, less likely to bail on the whole dump if something odd happens
  • dumper should recognise duplicate file errors and recover the fail
  • if the user has idled for the last twenty minutes, the db checks to see whether it can optimise itself
  • fixed flickering on fullscreen drag
  • fixed flickering onn fullscreen image init, meaning smoother mousewheel/whatever scrolling
  • fixed a graphical glitch in manage boorus dialog, and a bunch of others (removed all staticboxsizers)
  • fixed a similar glitch in manage ratings dialog - it now has dropdowns; I'll probably make it radio buttons again soon
  • fullscreen page navigation (previous, next, first, last) shortcuts are now customisable
  • custom filter now supports these new page nav shortcuts
  • can now do home/end and page up/down in shortcuts
  • shortcuts listctrls now sort by action by default
  • I improved the my custom listctrl's code generally
  • little less ugly code, here and there
  • fixed a whole ton of regular little bugs

• version 56

  • new existing tag sort dropdown
  • an option to set default existing tag sort
  • dumper error logic improved
  • dumper now does random sort properly!
  • double buffering hilariousness
  • some reordering of the local options dialog
  • made a 'shortcut' entry object
  • made a parent frame object
  • collection thumbnails will redraw if the sort changes
  • add media and add thumbnail data and graphics logic is improved
  • 4chan pass now authenticates for an entire year
  • bit more in help about the autocomplete tag entry
  • tag petition processing speed is a loooot faster

• version 55

  • custom filter prototype ready!
  • added some info on custom filter to help (in advanced.html)
  • you can now set a default tag service in options
  • tag rexeg dialog's listctrl refreshes without scrolling up to the top every time
  • tag for this file in tag regex dialog applies to all selected, not just the first
  • scrolling in autocomplete entry will scroll services in manage tags and tags regex dialog
  • ctrl+scrolling in autocomplete entry will scroll through results
  • complete rewrite of image prefetch calculation
  • couple of content-update display bugs in ratings filter fixed
  • in manage ratings dialog, sliders will preset to best guess at current score
  • dumper should be fixed
  • resized active predicates box (much smaller)
  • a deprecated call updated
  • added some forum extensions

• version 54

  • several new flash key and mouse event bugs fixed - now: mouse in = to flash | mouse out = to frame
  • the problem with scrollable management windows not drawing things on scroll is fixed; staticboxsizers now replaced with "- title -"
  • ratings filter now accepts several shortcut keys, directing to whichever window the mouse is over
  • the ratings filter windows will redraw themselves appropriately on content and service updates, e.g. changing tags with F3
  • ratings filter will choose the files to compare to more intelligently, both from the db and the currently-rated pool
  • the way services are identified behind the scenes is entirely overhauled and improved
  • the review service dialog is better as a result
  • all dialogs will now cancel-exit on pressing escape
  • started fade in and out for thumbnails
  • up cursor arrow in filter now skips
  • the tags regex dialog is refitted so it will refit on small screens better
  • dumper now dumps in its own thread (no longer locks up the gui while uploading)
  • the calculation of num_frames for flvs was off by a thousand, lol
  • synchro/wait shortcut typo fixed
  • inbox and other icons moved to background in fullscreen
  • fullscreen makes less db queries r.e. ratings
  • a couple managementpanel resize issues corrected
  • a couple drawing/flickering issues corrected
  • some updates to help
  • I made a forum to my site, and added links in the client and help

• version 53

  • several terrible ratings filter bugs fixed
  • ratings filter can now rate files internally
  • ratings filter frame now has a statusbar
  • system:rating:service=uncertain added
  • upgraded to wx2.9, which included a ton of bug fixes
  • sort by namespace wasn't working due to an odd typo
  • collect by logic changed a little so that 'none' values are collected
  • the shutdown lag from v52 is corrected

• version 52

  • ratings filter prototype is done
  • bit of help docs
  • more intelligent cpu burn on shutdown
  • dumper can manage flood time error, and recover from it
  • whole load of work done on mappings optimisation, meaning mappings updates, uploading mappings and approving petitions work about 100x faster!
  • forgetting pending uploads now throws up an ok/cancel dialog
  • had another look at overwriting deleted tags

• version 51

  • dumper deals with failed uploads better, although there is more work to do.
  • flv support for client and file repo!
  • reworked tag services' permissions calculations on thumbnail menu
  • new fatten A/C cache on shutdown
  • system:untagged works again for null tag repo ('all known tags')
  • advanced import options now default no limit for size and resolution
  • ratings service now have a little info on the review services frame
  • new shortcut keys (ctrl-b and -n) for animated gif previous/next frame in fullscreen
  • added thoughts on plurals to tag schema help page
  • hentai foundry downloads now interleave pictures and scraps
  • removed db locks on file/thumbnail reads
  • you can open the page picker frame with the mouse by going view->pick a new page
  • the copy->whatever thumbnail menu is more concise
  • system:hash now works for null file repos ('all known files')
  • there is now a title: regex on the tag regex dialog
  • servers now store updates in a folder, not inside their dbs.
  • couple misc changes to help pages

• version 50

  • 4chan pass added
  • captcha timings in dump are better
  • dumper's file ordering is less insane
  • some general updates to regex tag dialog
  • new box on regex tag dialog for tags for single files
  • regex tag dialog now supports arrow key up and down to change repo
  • entirely new and vastly improved listctrl class
  • collect by is now a thousand times more complicated, but also much more powerful
  • the top-left media-flicker display bug in fullscreen is fixed
  • some logging improvements
  • some tag cleaning improvements
  • thumbnails have less cluttered tag headers
  • some small changes to help
  • fullscreen precache doesn't do gifs any more, which should make things smoother
  • got rid of the awful view pending uploads dialog. I'll improve it later
  • started remote ratings, but its not done yet

• version 49

  • local rating like service
  • local rating numerical service
  • ratings dialog
  • ratings drawn on preview and fullscreen canvas
  • search by rating
  • system:rating in options
  • sort by rating
  • a ratings help page added
  • custom shortcuts
  • overhauled shortcuts code
  • a new, comprehensive regex menu in the regex tag dialog
  • extensive new hentai foundry search and sort options
  • a new option regarding the search style of tag A/C entries
  • thu dumper can now dump with tags
  • regex tag dialog selects tag A/C on page change
  • regex tag dialog's button now now shows in import dialog even if there are no remote tag dialogs with POST privilege
  • A/C is slightly faster in all cases
  • preview canvas now updates on a content update
  • numtags = 0 and numtags > 0 now work for pending tags
  • shift- or Ctrl-Click no longer deselects
  • listbook renaming and resizing bug fixed
  • collections thumbnail is fixed
  • sometimes, an empty tag, '', could be entered into pending and screw a lot up
  • sometimes, a null tag could be entered into pending and screw a lot up
  • advanced tag options has better checkbox defaults
  • some help screenshots updated
  • the db's processing is slightly rebalanced
  • the db's startup is slightly rebalanced
  • a misc gui bug was fixed
  • a misc gui typo was fixed

• version 48

  • New copy path and copy url menu entries on thumbnail right-click menu
  • Gif scrollbar
  • Hentai Foundry parsing errors fixed
  • Hentai Foundry 404 errors fixed
  • All the html parsers are rewritten using BeautifulSoap
  • Flash now zooms full, but not fit
  • Filter is now F12
  • Help is updated in a number of ways
  • Up/Down on empty input to change repository in manage tags dialog is fixed
  • The servers now support OPTIONS requests
  • Several status typos fixed
  • I started local ratings services

• version 47

  • Hentai Foundry interface is done, for both artist and tag searches
  • A/C gui reorganisation
  • A/C file and repo buttons added
  • A/C cache count alterations to match those buttons
  • A/C tag processing is cleaned up
  • Search and thumbnail-action improved to support null file service or non-null tag service
  • system:numtags=0 and system:numtags>0 optimised to be much faster
  • Ctrl+I now switches searching immediately/waiting
  • z now switches between zoom full and zoom fit in fullscreen view
  • advanced import options now has exclude already deleted files checkbox
  • Thumbnails choose the way they access the db more intelligently
  • ManagementPanel colour is changed, hmmm
  • Some DB cleanup
  • Some general code cleanup
  • Some download logic improved, including timer
  • Some download status improved
  • Download 404 notices better
  • DeviantArt now initially shows 'artist username' in its searchbox
  • Some code nomenclature improved
  • I've made a changelog.html, compling all changes together.

• version 46

  • DA parsing CLEAR
  • rule34@paheal parsing CLEAR
  • tbib (another booru) parsing CLEAR
  • Canvas rearranging
  • Preview now shows tags
  • The autocomplete tag write dialog will promote existing exact matches to the top of the list, even if it is not the most popular.
  • Autocomplete lists now scroll with the mouse.
  • A/C scrolling is a little more accurate.
  • Pressing up or down on an empty A/C entry will change tag repository.
  • A sledgehammer has been applied to the python garbage collector.
  • The code that navigates boorus is better.
  • The import queuing/pausing/cancelling system is better.
  • Booru tag-processing is better.
  • The import-queue gui is slightly better sized on small screens.
  • The new local tags service now shows up in advanced tag options.
  • Even more flicker is removed from the main thumbnail view, this time when clicking on whitespace. (it was a stupid bug)
  • Serverside db-backup now includes the WAL file.
  • Manually adding an admin service is fixed.

• version 45

  • Local tags: DONE
  • Thread watcher: DONE
  • Fullscreen now shows tags
  • Preview shows some more file + tag info
  • Listbooks load their panels on demand.
  • Local options dialog is now a listbook.
  • Some incorrect 'sample image' downloads from danbooru are fixed. Danbooru's inner workings remain a 404-cluttered mess though.
  • Some remaining utf-8 html parse errors (where a limited content-type header is given) are cleared up.
  • Another connection problem (a certain case of the server closing the connection) is now properly recovered from.
  • Some flicker in the thumbnail view is reduced, but my slow laptop still shows some, so I need to revisit this.
  • Autocomplete tag boxes now have little tooltips better describing their search domains.

• version 44

  • All database triggers are removed and replaced with more efficient inline code.
  • Cached number of files, namespaces and tags for a tag repo are kept updated, so review services should rarely take very long to load.
  • Cached number of A/C counts (current and pending) are kept updated, so A/C results should rarely take very long to load.
  • The way A/C cache searches are done is improved.
  • The way A/C cache entries are calculated is vastly improved.
  • The way several db SELECT DISTINCT * and SELECT COUNT( DISTINCT * ) queries are performed is improved.
  • The custom httpconnection object is improved to better deal with server-side connection closes. (Cannot connect to server problem)
  • The custom httpconnection object does http redirections more intelligently.
  • The custom httpconnection object now parses non-ascii content-type (e.g. utf-8) properly, according to server-side header. The tag "pokémon" will now parse correctly.
  • e621 and rule34@booru.org are fixed. (just a couple parsing-variable changes)
  • If some tag parsing entries are checked on a booru download page, the tags will be downloaded, parsed and applied even to files already in the database.
  • The mouse cursor will remain visible in the fullscreen browser while the right click menu is open.
  • I corrected some which->that grammar in a couple dialogs.
  • Auto-repo-setup (from the help menu) was broken in a very stupid way, but I fixed it. I apologise for not noticing.
  • Thumbnails now show creator and title up top. They are a bit crushed, so I may rearrange them.

• version 43

  • Updated import + booru code
  • New booru buttons
  • Cut tag parse bug fixed
  • Download preview instead of real file bug fixed (I think)
  • Fixed the 'thumbnails not appearing on import' bug
  • Import flickers less
  • Media Classes rewrite is done, although I'm not totally happy. Have to think about it.
  • A problem with pending tags applying over existing tags no flushing correctly is fixed.
  • A/C cache is better, but still a little slow on slow computers.
  • system:duration=0 is fixed.

• version 42

  • Autocomplete tag cache DONE. Please tell me if you encounter excessive slowdown or incorrect counts.
  • Autocomplete tag logic improved in about five ways. Tags should appear more intuitively and with more accurate counts.
  • Copy and paste buttons in manage tag dialog to facilitate spamming many tags from one file to another.
  • Imageboard dump is fixed!
  • URL download got fixed last week and I forgot to say!
  • A peculiar pseudo-foreign-key bug with the active_pending_mappings table is sorted. (leading to more accurate queries and counts)
  • Since display of large queries is so much faster, the 'woah! 12,000 query results' warning is gone.
  • Fullscreen precache timings are rebalanced, hopefully for the better!
  • The importing gui classes are a little smoother, and thumbs should load one by one, not row by row, once the page height is exceeded. The scrollbars should be less janky.
  • Page transitions are smoother.
  • Importing from URL will now load those urls that are already inside the db (hella fast, too), just like importing from hdd.
  • The double-update-download bug is finally fixed! Now you'll download update number 258 only once, and not twice in a row. (I hope!) It never mattered much, beyond a little CPU, but it annoyed the hell out of me. Working out what was wrong took ages. (Long story short: Python's Priority Queues are not parallel FIFOs!)
  • There was a weird bug in pending tags sometimes going to higher tag-precedence services, rather than the one chosen in the manage tags dialog.
  • There was a typo in reset service.
  • More content update unification in the db, making things much neater behind the scenes.
  • I found a potential hang in the db-shutdown code.
  • I found another potential hang in the db-shutdown code!
  • The number of messages in your inbox is now displayed on the statusbar.
  • Bold, Italic and Underline now work with Ctrl+B/I/U in the message compose panel.
  • The 'don't receive messages from Anonymous' checkbox in Add, Remove and Edit Services for message depots now does something.
  • system:unread in message search now has a count.
  • A draft will remember its saved changes when you click on another conversation and back again.
  • You can now save a draft with zero recipients.

• version 41

  • Rich Text Control with Toolbar in messaging.
  • Message unread/read status now syncs to message depot.
  • Can set defaults for file system predicates.
  • Animated Text class written.
  • system:unread tag in messaging.
  • Import bug fixed.
  • Messaging search logic improved.
  • Message depot default check time down to 180, and daemon thread improved to manage with that.
  • Application startup is more stable.
  • Some AddMessage logic improved.
  • Drag+Drop service fixed and improved.
  • Tags added while importing will now turn up in autocomplete.
  • I unified more of the client's content-update db calls. Just making stuff neater.
  • I massively simplified the autocomplete tag code in preparation for another bash at doing autocomplete cache. A/C is far too slow atm, imo.
  • Some A/C counts are still off, particularly when there are a mix of current and pending. I couldn't figure out what is causing it, but I'll look at it more as I do the cache.
  • In add, remove and edit serivces, you can now specify if you want to receive messages from Anonymous or not. Unfortunately, because I forgot to implement it in DoMessageQuery, this checkbox doesn't actually do anything yet! Whoops!

• version 40

  • Inbox/Archive for messaging works. The inbox icon is back, and F7 to archive as usual.
  • Unread/Read for messaging works. Click on your status next to a message panel to change.
  • All message participants are shown in the conversation listctrl.
  • The message page will open with 'system:inbox' active, ordered by newest message.
  • I fixed conversation listctrl sorting in a couple ways.
  • Some bugs in the db->gui update reporting system are fixed, so the compose window and draft panel will close properly on a message-send, and the conversation listctrl will swiftly update unread counts and inbox status and so on.
  • The draft panel's save button wasn't updating sometimes when it should have been.
  • html links in messages will show in the main frame's statusbar and launch in your browser when clicked.
  • I reworked how many system tags are searched in file queries to make them muuuuch faster. Now, any time you go system:hash, :size, :mime, :age, :width, :height, :duration, it (should) be significantly faster than before.
  • Ever have a png file that had an ok thumbnail but a black preview/fullscreen? It was probably an 'LA' image, or 'greyscale with an alpha channel'. Display of LA images is now fixed.
  • Streamlined the way file 'archive' and 'delete' commands are sent throughout the application, with an eye to eventually adding all other user-commands to the same system (to make it simpler, and also to do undo-redo).
  • I've updated the help docs, including the section on how messaging actually works.
  • You can now enter system tags with enter/return from the keyboard, not just double-clicking with the mouse.

• version 39

  • A whole bunch of stuff I didn't write down that generally made messages work beyond sending "Hello World".
  • Unicode messages now work.
  • Newlines in messages now display properly.
  • Fixed contact saving and drag-and-drop to the manage contacts dialog.
  • You can now add a contact using a contact address (assuming their service is v39).
  • A bug in contact renaming is fixed.
  • My contact (hydrus admin) is now added automatically.
  • Messages are orphan-deleted properly server-side.
  • Message depots you control now get an entry under the admin menu.
  • The conversation listctrl is now properly sorted.
  • The HtmlWindow that shows message body is now resized to its minimum needed height via magic that I ain't gotta explain.
  • Messages that fail to send can be retried by clicking on the failed.
  • Some draft save/send button enable/disable logic was updated.
  • I switched around some server-side account logic.
  • You can't zoom a flash window bigger than the screen.
  • I've made the mouse appear in fullscreen when you move it, and this includes while you drag. See how you feel about it. I'll make options to change behaviour if people care.
  • Better zip structure. (No more CWD rubbish)
  • New tar.gz release with only the code and help. Code is 1.2MB, help is 14MB lol. I'll improve the structure of this release next week. I threw it together by hand today, but I'll improve its structure next week.

• version 38

  • messaging v1.0

• version 37

  • messaging prototype

• version 36

  • animated gif fixes
  • new page chooser
  • svcp->cstvcp in fullscreen too
  • new import progress
  • just start the server on this computer, please
  • I think I have scrolling to the current selection after an archive fixed now.
  • Started an experimental simpler way to send commands to the db, which may open the way for undo-redo.
  • Capitalisation option in file->options->gui. It only does the menubar for now, but I will expand it.
  • Custom sorts/collects. Under file->options. Check it out, play with it if you care about it.
  • Some program initialisation error handling is cleaned up.
  • Can now no longer start a server if the admin port it wants to use is occupied. (can't run two servers at once on the same port)
  • I broke dump with a typo in a previous release, I think! It is fixed now.

• version 35

  • Slight speedup of review services frame loadtime.
  • SVCP now replaced with SCTVCP and CSTVCP for now, with more planned next week.
  • Repo sync daemon adds db jobs less manically, now, locking up the gui less.
  • Repo sync daemon reports successful updates to the log.
  • Tag petition review page has a visual bug fixed.
  • ThumbnailMediaPanel manages its underlying bmp canvas more intelligently.
  • Some ugly page-closing code was cleaned up.
  • There is a 'regex examples' popout window on the regex-filename-tag import dialog, with copy-pastable examples of python regexes that'll work for common filename-parsing situations.
  • Home and End keys now select the first/last thumbnail as well as scrolling.

• version 34

  • DB consolidation, and foreign keys and triggers with it. The db is less prone to orphan ids and miscounts.
  • For long-time users, all orphan ids should have all been deleted, reducing weirdness.
  • Total size of local service is now tracked accurately.
  • Archive while importing should work a bit better now.
  • Update logic should be more reliable, as should all service_updates.
  • Archive count for file repos should be less wrong.
  • Fixed two (!) typos in system:age predicate code.
  • File upload is fixed (not sure if it was ever broken in release, but I fixed the stupid thing I did anyway)
  • DB (and server) errors should report better now, revealing inner-db code in the traceback. (and printing it to the appropriate log)
  • Improved the mime_to_enum object for image/jpeg, image/jpg, image/jpe, image/wtfaretheygoingtocalljpegnow.
  • HTTP connections can deal with some esoteric commands, like 301, without throwing a fit. All http goes through the same code.
  • Danbooru should now be in your booru list, and it _should_ work.
  • Pausing/Cancelling a booru gallery parse should now work.
  • 4chan dumper now understands captchas being wrong and the thread not existing; in the former case it'll let you put in a new captcha and try again, in the latter it'll stop the whole dump.
  • Some general code neatness improvements.

• version 33

  • Boorus are working for users who started last week
  • The server can now manage many more simultaneous requests without bugging out.
  • The early freeze/import freeze due to simultaneous access in the client is fixed.
  • The client's file+thumbnail server (details in help's advanced.html) works again.
  • Client DB now stores md5 and sha1 for all local files.
  • Client DB has a larger cache size.
  • The freeze caused when importing thousands of files at once with tags is fixed.
  • The yaml imageboard import (when you drag and drop a .yaml file onto the imageboards dialog) now adds form fields correctly! If you had problems here, check out install_path/static/default_imageboards.yaml to reset them!
  • The autocomplete counts should be more accurate, particularly when there are a mix of current and pending tags in the db.
  • The autocomplete in the regex-path dialog will now search the db correctly (it was misconfigured).
  • Autocomplete is also just a _little_ faster
  • File imports are now low priority, so gui activity should still happen nice and quick even while an import is occurring.
  • The review services frame will now show how long until the next update will be checked (see picture above). In case of error, it will show how long until it tries again.
  • The review services panel and manage services panel are of a better default size, and they use my new listbook class exclusively, rather than the old notebook.
  • I've added some tooltips on not-obvious widgets.
  • Content Updates have vastly improved processing speed on a thumbnail view (x100 in some cases!). Tag updates particularly will not be so slow to process if you have a large search open.
  • The import gauge will now show the correct range according to the current queue size, not always the default 100.
  • Active Import Pages will throw up a yes/no dialog if you try to close them.
  • Some beautifying of code syntax. '\n' is now os.linesep, which means exported stuff should show up better in notepad.
  • Some changes in network protocol, although now I think of them, I realise I forgot to update the network version. They aren't important for most users, so w/e.

• version 32

  • client db serialisation DONE
  • Cleaned up a dirty-bmp canvas init issue.
  • Resized some dlg stuff that was too tall
  • Cleaned up a bunch of unrelated db code
  • Tightened the pubsub system so it integrates better with wx.

• version 31

  • booru prototype
  • Fixed some problem with last error timestamp
  • Fixed a bug in repo bandwidth tracking
  • Fixed URL registration with the client db.
  • Made downloading URLs asynchronous again
  • Made repo synchronisations not block so much, and not for nearly as long. You should be able to do queries and click on thumbs (albeit at a slower rate) while the client is downloading and processing updates.

• version 30

  • Thumbs are exported, and, along with perceptual hashes, now treated as a global pool, reducing downloads and speeding queries.
  • Thumbnail resize is now asynchronous, done as each thumbnail is requested. Changing the thumbnail size in the options dialog happens real quick, and if you don't like a particular size, you can change it back just as fast.
  • Temporary DB locking mechanisms ensure serialisation, less inefficient locking.
  • ImportPage superclass works a lot better, in a lot less code. DB import code is now even simpler, and much faster, especially for redundant files. The new import page will also tell you how many files were successful/failed/already deleted/already in the db.
  • Fetching images for fullscreen or preview is fully asynchronous, and will no longer block the gui thread.
  • Some query logic is improved.
  • You can now import and export sites/imageboards from the manage imageboards dialog. It'll export to a .yaml file, which is easy to understand with a decent text editor. Just drag and drop onto the dialog to import.
  • Selecting thumbs has a little less preview-flicker now.
  • The new tagboxes now render numbers prettily, so 4123 -> 4,123.
  • (+) namespace:tag now colours properly.
  • The background for selected tags is now an appropriate colour.
  • You can now choose the colours of any namespace you like in the normal options dialog.
  • Clicking in whitespace in the new tagbox deselects, rather than select( 0 )s.
  • Spacebar, PageDown, Return and all the other special command characters are now propagated to flash in fullscreen view, just as long as the mouse is over the canvas.

• version 29

  • Big changes to the db's file storage, as above.
  • database->vacuum is reinstated, database->reindex is retired. I recommend you run a vacuum once you are updated; it'll only take a minute now.
  • Rollback and initialisation bugs are fixed.
  • There's a splash window now that can offer client-boot feedback.
  • The way scrolling-to-focus is calculated is improved.
  • Pretty much everywhere you see tags will now show coloured tags! I finished up my own tagbox class, which for now will show tags coloured in the default booru scheme. It also loads in a few milliseconds, rather than the ~0.5s of the default listbox. It scrolls faster, too. I'll have user-customisable colours next release.
  • Files with invalid resolution, size and mime are now filtered out before dumping to an imageboard.
  • A number of misc other bugs were fixed.

• version 28

  • I fixed the 'searching immediately' bug. It was a stupid typo as usual.
  • Post-Filter scrolling should be a bit more reliable.
  • Some more log events.
  • I did a lot of work on my new tags box, although the actualy gui elements will have to be finished for v29.
  • Manage Imageboards wasn't saving properly. It is now.
  • My new progressdialog happens in several places. It is a little neater than the one in wx; just better for my purposes.
  • I reworked some of the transaction stuff in the db so that I can atomically queue pubsub events. (this means file import are less juddery)
  • After becoming very annoyed with wx's listbook/labelbook/whateverbook, I wrote my own. It's all over the place in my dialogs now. It is tighter, more reliable, and can scroll.
  • I reworked some of the way sets interacted.
  • There was an infinite recursion bug sometimes, when uploading files to a file repo.

• version 27

  • Cleared out some old unused library imports, so the client is slightly smaller.
  • Had a cyclical import (!). Luckily, python is forgiving, but I have still fixed it.
  • Collection thumbnails will now update with their new pending tags correctly.
  • Some wx.Panel vs wx.Window issues and nomenclature resolved.
  • Full Page decouple. Now the window to the left is a seperate, much neater class. It and the media/preview windows talk to each other in a clever way that means I can make new kinds of page much faster. It scrolls, too.
  • Thumbnails resize will tell the thumbnail window just to redraw itself, not refresh the entire query.
  • 'system:local' and 'system:not local' are now mutually exclusive (will swap each other out, rather than overlap), like inbox and archive.
  • A little neatening of the system predicate code.
  • 'system:similar_to' RMB menu entry fixed for some cases.
  • There is a new log window. Go view->new log page to see it. It doesn't show much right now, but you can see synchro successes and errors pop up as they happen.
  • There is now a popup button on the import dialog and the URL downloader page that does 'advanced import options'. You can now tell these import processes to archive everything and/or ignore files below a certain size or resolution. By default, the client will not import anything <5KB or <50x50px. You can set no limit.
  • Flash windows will now own all keystrokes as long as the mouse cursor is over the flash. If you are browsing/filtering, just move your mouse over or off the flash window to make sure your return/page down/spacebar/escape/whatever is interpreted as you want.
  • The URL downloader will not download the same file twice in a row, so you can add a thread URL several times in a row, and it'll only download the newest files. The behaviour of this is affected by the 'exclude deleted files' checkbox in the client's options. I'll implement thread watching a bit later.

• version 26

  • The double unnamespaced count bug ( 'system:blah (70)', 'blah (140)' ) in autocomplete should be gone.
  • Some rejiggering of how the statusbar updates means active import windows can't spam the statusbar when they don't have focus.
  • The count of pending tags is more accurate and uniform. When it says (+3), it means for three files, not in three cases.
  • Tag counts overall are better. When it says 'diaper vore (17)', you should be getting 17 files if you enter that search.
  • GUI tag repo precedence is slightly more accurate, compared to what the db thinks it should be.
  • I nailed down why system:mime wasn't working sometimes, and now it should work all the time.
  • I discovered some problems with repo-side bandwidth tracking. I fixed a couple bits, but have notes to look at it more.
  • In v25, if system:limit was greater than the search population, it threw an error! Whoops! Fixed now.
  • The post-filtering thumbnail problem is totally fixed, as are a couple other instances when thumbs would sometimes not draw.
  • The fullscreen change-image-jittering-around problem is fixed, I think.
  • Fullscreen archive works again. Not sure why/when it broke.
  • Importing with archive checkbox ticked sometimes didn't archive, lol.
  • A bunch of decoupling of gui elements, making for much neater code and less gui blocking.
  • Decoupling of import from hdd as well.
  • Query is now asynchronous! If you do a massive query on a slow computer, you won't lose the gui! You can go check other tabs, whatever, and the query will process in the background, rendering on the page you started it when it is done.
  • A number of cpu-heavy numbers are now cleverly cached, so system predicate fetch happens much faster, especially on 100,000+ image dbs.
  • Some 'collect' code is sped up and made generally less bonkers.
  • I wrote a couple new classes for my most common gui elements.
  • There is now an option in file->options to change the number of characters before autocomplete will go search the db for tags! If you have a slow computer and a huge db, you can set it to 2, 3 or even more chars. Default is 1.
  • Dialogs that ask you questions will now usually center on their parent window.
  • The current selection tagsbox won't show ( 'blah (12)', 'blah (+3)' ), it'll combine into 'blah (12) (+3) (-1)'. With the + meaning pending and the - meaning petitioned.
  • Autocomplete now respects which of the include_current and include_pending buttons are green! It'll only search current/pending as you wish! And the numbers will now be accurate!
  • My new question dialog with the yes/no buttons hooks the Escape key to mean no.
  • I don't use it much yet, but I have a new message dialog too.

• version 25

  • Thumbnails should load just a bit quicker.
  • Made some dict + set code neater
  • A new end-of-filtering dialog makes the choices a bit more obvious
  • When you exit from fullscreen/filter view, the main thumbnail view it was based on should select (and scroll to!) where you were. But filtering is less reliable, since you may have just deleted the last thing you were looking at!
  • Better scrolling in thumbnail view
  • Scrolling in thumbnail view will follow the current selection, so the focussed media will always be visible.
  • Double right-clicking in filter now deletes instead of keeps (typo)
  • Slightly less flickering when processing repo updates
  • Made some form code neater
  • A lot of the thumbnail gui code has been decoupled and pubsub'd. It is all neater behind the scenes, meaning I'll be able to extend it easier in future.
  • I completely overhauled how my pubsub works. (pubsub is a messaging system inside the code). It is more flexible and easier for me to work with.
  • system:mime was not working in v23, not sure why. It's fixed now.
  • Some small speedup + neatening in db queries.
  • You can now scroll to the end of a thumbnail page by pressing End. WARNING: If you have like 10,000 thumbs in the current pane, it is going to have to load all those.
  • In the AutocompleteTagWrite object, which is the tag entry box in the manage tags dialog, there was a stupid bug that made many requests more expensive. It's fixed.
  • Autocomplete now works off one character's input.
  • I've swapped a bunch of expensive objects with easier staticbitmaps for now
  • some A/C ordering changes

• version 24

  • imageboard dumper prototype
  • some db update stuff

• version 23

  • A typo was disallowing petition rescinds in certain cases.
  • I've rearranged some of the RMB menu separators to reduce filter/archive misclicks.
  • Tag pending upload has more accurate GUI response.
  • You can now find similar images from RMB menu.
  • Petition approvals are mirrored on an admin's client.
  • The gui now accepts that admins can overrule deleted content.
  • Couple tiny URL downloader gui bugs fixed

• version 22

  • system:similar_to
  • Some backend namespace stuff
  • Couple of gui presentation fixes
  • I removed the clunky inbox and archive buttons, see how you feel about it.
  • I tried to do ultimatelistctrl to have coloured tags (like a *booru), but it was too buggy and slow for the client's purposes, so I'll probably have to write my own control.
  • system:mime exists, now. Not many mimes, yet, but have a play with it, see if you like it.

• version 21

  • On opening the URL downloader, the textbox input focuses.
  • The URL downloader has some better update texts. It pauses nicely, etc…
  • The URL downloader now orders urls in the order they are parsed. I forgot I was throwing the links in a set, losing all order!
  • You won't see dupes in the URL downloaders' mediaresultswindow.
  • Accounts should now correctly refresh on version update.
  • When accounts fail to sync, it dumps the error in the log.

• version 20

  • The preview pane should be a little less schizophrenic.
  • Some code cleaning up behind the scenes.
  • Thumbnail download optimisation if they already exist locally.
  • Icons will show/hide in fullscreen with mouse movement, just like 21/123.
  • Weird archiving bug on import is fixed. It was display only.
  • Manage tags dialog now works in fullscreen, lol.
  • system:not_uploaded_to: now works
  • services->review services now loads muuuuuch faster (after the first time)
  • There was a typo in setting up initial server's accounts. Email me if you can't make new admins.
  • Importing is more streamlined, with more shared code.
  • Importing with tags dialog happens more naturally in the import process. You can now do it from drag and drop imports!
  • I reordered the default collection dropdown. Your default may be changed, so check your options.
  • You can now fetch an account by a deleted file (hash).
  • Thumbnails' tags now obey tag precedence for deleted stuff. I hope.
  • Modify Account Dialog now does its stuff.
  • Collections thumbnails now show how many files are in the collection, just next to the icon.
  • I've added to and edited the help extensively, replacing screenshots and updating the text.

• version 19

  • drop librarium nomenclature
  • More intelligent prefetch in fullscreen - things should load muuuuuch faster
  • A little more intelligent sum() calcs in autocomplete - non-namespaced tags now sum all their namespaced versions as well
  • More rigorous client db transactions, just to make things more pretty and reliable.
  • Moved db files to /db, logs to /logs
  • Made inroads to getting review services to load faster, but version 20 will have more here.
  • I've updated the frame view so the file info is draw on top of the image. It's a hack for now, inside the image-frame, but it is the only solution for wx's annoying 'no overlapping' policy.

• version 18

  • Some misc gui image info bugs fixed
  • Review services dialog is less pubsub buggy
  • Fullscreen image prefetch does more and hopefully more intelligently
  • Entering a blank tag to quit tag dialog was broke when I updated autocomplete textctrls. I've fixed it.
  • Petition page is fixed, so I can finally clear out the tag petitions!
  • F10 was broke when the lib had deleted files lol! Fixed it.
  • Review services is more responsive to repo permissions updates.
  • Flash files are now correctly put in the inbox on import.
  • Synchronise Repositories daemon no longer takes ten gui-blocking seconds to process every update! This was an annoying slowdown, but I figured out what it was and reduced it to <1s with a quick patch. The problem will return with private tag repos, but I'll deal with that when I come to it.
  • Synchronise Accounts daemon has some better pubsub so it fires more intelligently. This in turn affects several other daemons, which should all be more responsive; e.g. when your permissions change, your pending thumbs/files should start downloading.
  • Improved delete orphans logic for the fiftieth time! It is great now, and won't ever bug you in the gui; it runs invisibly on shutdown and usually only takes a few secs to do its job.
  • Some of the gui updates triggered by the daemons are optimised so they block the gui less. Multiple stuff is bundled into single updates, those updates are less buggy, and the gui catches them better. Resetting the public tag repo will no longer lock up a slow system.
  • A few versions ago I changed the way repos were registered with the gui, and I've fixed some of the resultant daemon-related errors. This mostly affects admins, which mostly means me!
  • Download Thumbnails daemon was getting pretty slow for large dbs (100k+ files) because of an inefficient db query. This has been cleared up.
  • I banged my head against the wall for a couple of hours getting nested notebooks to work, and I am now a better person for it. Review and Manage Services dialogs feature better panel management as a result. Their panels are also much richer, reporting loads more data about the respective service. You can now see how many mappings the public tag repo has without having to delve into sqlite db browser.
  • I have unified panel and frame background colouring away from wx's bonkers idea of what things should be coloured. It should mean that everything is properly grey, but please tell me if your gui turns all black or transparent or whatever.
  • 21457 is now 21,457 in pretty much all cases where it is appropriate.
  • Fixed a bug with wx gauges not liking large ranges because of signed short or whatever underlying var the C code was using to store it. (That should have been done at the wx-python level, but whatever.)
  • 'localhost' resolves to 127.0.0.1 when editing a service, mainly because windows freaks out at the former, deciding to take a couple seconds longer to make any raw connection.
  • Dialogs and frames have better, multi-monitor-friendly positioning.
  • Flash files are loaded from a temp folder beneath the install dir, rather than lib's server, for various reasons. This temp dir also manages file copy/paste operations. I may move this back to the OS's approved temp folder, but we'll see.

• version 17

  • Tag negation "-tag" works again
  • You can now cancel a filter on the first image
  • Some weird filenames that were breaking the new import page no longer do
  • Autocomplete (either when searching for tags or adding new ones in) will now show the number of files tagged with that tag after the tag e.g. here
  • Manage tag dialog now pops up slightly to the upper left, rather than the center (default) of the screen, so you can see what you are tagging. I may make this more intelligent in future, so it remembers where you last left it.
  • I changed a couple small things in help regarding the new import.

• version 16

  • Numtags system predicate now takes into account which of current/pending tags buttons you have active.
  • Thumbnail downloads fire more reliably.
  • Downloading a file was causing funky thumbnail draw, which seems to be fixed.
  • Delete Orphans daemon only fires on application close now, so post-filter db lag is gone.
  • Daemons now use events, not conditions! lololol, I'm an idiot who can't scroll down a library doc. For the longest time I was like "I wish 'threading' had events!" and it does.
  • Some account checks were missing in tag upload, but are now there.
  • A gigantic db rewrite that altered pretty much every table and query.
  • Calculation of tags for current selection now support collections, and is fast.
  • Was deleting a repo broke in 15? If it was, sorry! Didn't catch it, but it's fixed with the new system now.
  • Pubsub can do more, and is a little less 'argh, I think it works like that', and more OOP
  • The way tags are held by a thumb can support multiple tag servers (but only does public tag repo for now)
  • Repos are now a subclass of 'service'. I'll do more on this in the next version, including some nomenclature changes in the gui.
  • Imports are now asynchronous! There's a new import page that'll open on any import. It's a little jittery, but it all works. If you import 3000 tiny files, your gui may lock up for a few secs since it'll be able to import faster than it updates the gui.

• version 15

  • manage tags dialog focus bug hotfix

• version 14

  • Selection's tags box now shows collections' singletons' tags
  • Autocomplete now sources from the current thumbnails whenever it can (faster than db), and does so intelligently
  • Couple of small display bugs fixed
  • Some delete logic approved
  • Repos do some more efficient inserts on POST content
  • Petition optimisation, for auto-soft-approval before update gen
  • Dialog parents, dialog parents everywhere
  • DB cleanup of a couple of old methods
  • Autocomplete Write (when adding tags in manage tags dialog) now sources from the entire network's tags (not just for the files on your hard drive) when you type enough letters to stump it. i.e. If you have no files tagged 'evangelion', but you do have a file tagged 'eve', it'll show 'eve' if you type 'ev', but if you type 'eva', it'll requery the db more widely, and give you 'evangelion'.
  • system:ratio for searching for wallpapers or whatever. Combine with system:width>x for extra utility!
  • You can now hit F3 (manage tags) in fullscreen, whether browsing or filtering.

• version 13

  • a couple of cpu-intensive DAEMONS schedule themselves better. (less slowdown after a big filter)
  • image rendering prefetch improved
  • smoother image rendering to screen (especially when zooming), less flickering
  • petition approval has smoother, less buggy workflow
  • new hashes and tags are added to the db muuuuuuuuch more efficiently. most public tag repo updates happen in a second or two, rather than 30+
  • some of the internal commands are streamlined together into a single class
  • ctrl-s fixed
  • clear pending file repos pubsub fixed
  • ctrl-r now shows/hides splitters
  • thumbnail delete logic is better
  • select logic is better
  • deleted_remote_files bug fixed
  • headache-inducing weakref repositories __eq__ bug in pubsub fixed
  • 'with' keyword all over the place now. turns out wx has a bunch of objects with support it (including dialogs!)
  • gridsizers are now all flexgridsizers
  • the list of tags on the current selection now has a weight attached like 'flower (23)' to show how many of the current selection have that tag. it doesn't support collections' weight yet.
  • revised some ambiguous repo error messages
  • you can now have a main window with no pages open
  • right click menus on a selection are vastly improved (lots more info), but still a way to go.
  • on the import files dialog, you can now specify to auto-archive rather than send to inbox
  • thumbnails now know where they are uploaded/pending/petitioned to and will show that info with different icons
  • escape key now deselects
  • manage repos dialog is completely revised. comments appreciated.

• version 12

  • You can now tell lib to search 'current tags' only, 'pending tags' only, or both.
  • Tag search is more efficient.
  • Search in general is more efficient.
  • Search code is more beautiful and decoupled.
  • Tags are stored in memory more beautifully, and a similar class holds repositories for thumbs also.
  • Age predicate is fixed (twice!)
  • Server code is more beautiful. DB-Server interface is decoupled into a mixin class to make it easier to understand and manage.
  • Icons for pending and petitioned files, and for 'currently downloading'.
  • Prefetch fixed, so nearby fullscreen images load just in case you want to see them.
  • Fixed a redundant pending tags bug
  • Fixed a swf tag petitions bug
  • On boot, temp folder is cleared.
  • The 'current tags' window is now union on the current selection, not intersect. If nothing is selected, it shows all the tags of the current selection. In a later version, I'll order this list by count-weight.
  • All (I think) tag updates are synchronous-ish with the gui. As updates come in from the public repo, thumbs will update themselves appropriately. Same goes for repositories.
  • Gui-code tidy-up
  • On deleting a local file, preview clears.

• version 11

  • swf support, lol
  • You can now backup repositories from the admin menu. Info in help.
  • Under the help menu, a quick way to set up the beta repositories' credentials.
  • Account sync was full of bugs, and is now less so.
  • When you go fullscreen, the preview window should go blank, which should save a little cpu time when you are looking at gifs and your ears when you are looking at swfs.
  • Auto-complete is muuuuuch faster than it was. Also more correct. Still more work to do, though.
  • Librarium hosts an http server on 45874, which supports GET /file and /thumbnail just like a file repo. Utterly prototype, but you can test it right in your browser if you like. Go http://127.0.0.1:45874/file?hash=whatever when lib is running.
  • Ctrl-T opens a new local page/tab, Ctrl-W closes the currently active
  • Image display in preview and fullscreen has been revamped, and is now both a little buggy and much better.
  • Bandwidth tracking both server- and client-side should be more reliable.
  • Delete lag (the slowdown after you boot/delete something) should be reduced.
  • Query lag is significantly reduced, and new operators for predicates have been added. You can now go ≈ on a bunch of values like size and width and duration to get 'approximately equal to', which actually means plus or minus about five percent.
  • A stupid error on file upload is fixed. If you've been having problems seeing the files you uploaded in preview, try resetting the respective file repository.
  • A few daemon bug fixes.

• version 10

  • hotfix for tag updates

• version 9

  • Pressing a cursor key when no thumb is focussed no longer throws an exception.
  • Hitting enter on a blank input in the manage tags dialog now quickly oks the dialog.
  • Annoying splitter exception on import is gone. It was a one-line fix, because I was being stupid.
  • All searches should be a little faster. Mostly effects searches with 1000+ results.
  • Modal dialogs (in particular) should be just a little more responsive now. Less stuff is being dumped on the wx gui thread, and all db handling is more parallel.
  • Slow petition approval fixed, made almost instant.
  • Image rendering has been revamped. It should be less stuttery now, but there is still more work to do.
  • Some pending tag update logic was fixed.
  • If the focused thumb goes from 'not local' to 'local', its preview will load. (its on my to-do to do the opposite: clearing preview on delete in remote search.
  • DB daemons rewritten completely, so now a whole bunch of stuff flushes more efficiently, more reliably, and with less gui-blocking. Things happen smoother now.
  • Can now create repository database backups via librarium. Sending the command (via the admin menu) tells the repo to lock and copy its .db files to .db.backup. I'll update help to explain this more.
  • Can now copy files from within librarium to the clipboard! Just select from the right-click menu or press Ctrl-C, and your selection will be copied to clipboard. You can then paste to any directory, for quick export to anywhere. The export command remains for exporting thousands of files.

• version 8

  • Fixed a missing transaction on repo db update.
  • Export path can now be on other hard drives. You'll probably want to keep it on the same drive if you have lib on a usb stick, though.
  • Fixed a really stupid '<' for '>' typo that was causing a ton of gui blocking and repository slowdown.
  • File and thumbnail downloads are much snappier.
  • One person had a very odd error while trying to open manage repos. I don't understand what actually caused the traceback to happen, but I think I fixed it nonetheless.
  • inbox/archive buttons above the searchbox are now mutually exclusive.
  • Fixed the version number in lib's about window.
  • Fixed the preview window so it remembers its last size properly. Also reduced its min size so you can make it tiny.
  • Fixed the problem people were having with admin rights. See this post if you were having problems.
  • The RegisterRepoRequest that was filling people's logs on upload should be fixed now. Turns out I didn't understand how wx.CallAfter works during modal dialogs! I threw in a horrible fix, but it works. I'll revisit it.

• version 7

  • db changes are done
  • fixed a bug with posting tag petitions
  • when typing tags in, autocomplete now harvests from pending tags as well as current
  • the regex dialog's tag entry is now autocomplete
  • leading/following whitespace removal for tags, and multi-whitespace -> single
  • a little server-side pubsub neatening up
  • when importing via dialog, there is now a checkbox to ignore subdirectories
  • Added a quick readme.txt in the base_dir for people who choose to extract rather than install. (Everyone should check out the help at least once)

• version 3->6 (never released)

  • after much feedback, completely rewrite db
  • general improvements to gui

• version 2

  • general fixes and improvements

• version 1

  • first true metadata beta
  • shit actually works

• versions 0.91 -> 0.99

  • move to python and wx, and an application- rather than browser-based model
  • building on concepts
  • scrapping p2p, turning to client-server

• early versions of beta

  • made db, playing around with LAMP
  • an initial p2p framework started