Startrails

[ova103]

Hello,

A question on startrails: In my stratrails I see as well day and night images, although angle has been set to -6 or some other days to -12.
You can see it here: see upper left corner where DAY and NIGHT appears

However, in the documentation I read that only images of the night will be used

So why would day images be used in my strartrail?

Thanks for the feedback

[EricClaeys]

@ova103,
This page explains how to set the Brightness Threshold setting, which is what determines which images are included in the startrails and which are not.

Let me know if it does NOT answer your question.
I believe everyone who's followed the suggestions has gotten a good startrails.

[ova103]

Hello, thans for the answer, but it does not answer the question. I follow the instructions on how to set the Brightness Threshold setting so that is not the point. My question is different: why are DAY images used for the startrail?? While the documentation (on top of the page which you have added as a link in your reaction) is saying that only Night images are used. Thanks.

[EricClaeys]

Startrails creation doesn't know which images are daytime and which are nighttime. It only looks at the overall brightness of an image.

You are right that the documentation page, strictly speaking, is wrong, but most people have the Threshold set low enough that only nighttime images are used. I couldn't find a way to concisely explain that on the documentation page, so chose the current wording instead. If you have better wording, let me know.

What did allsky-config get_brightness_info output? Can you attach a screenshot?
What is your Brightness Threshold currently set to?

[ova103]

ok clear. You could just add to the documentation that all saved images (day and night) are used to make startrail.

allsky-config get_brightness_info: --> unknown command. I still use an allsky version from Dec 2024 or Jan 2025. Reason I did not install more recent version is that I had to do some tricks to get the images on a SSD (while still booting on a SD - maybe you remember that). I know that you described in the installation how to do, but as my systtem is working right now so i'll keep this version for now.

I re-run generated the startrail for 10/7 with generateForDay.sh --debug --startrails 20250710
and this is what I got:

Treshold 0,20
startrails: Minimum: 0.179434 maximum: 0.782581 mean: 0.391017 median: 0.350245
only day images with clouds
images of the night seems to be brighter so not on the pic:

Treshold 0.25
some images of the night, but still missing more the more brighter images of the night

Treshold 0,275
good startrails but with the darker day images.

So the issue is that the day images seems to be darker than the night images. Did you see that kind of problem before?
(and so this was the reason I was hoping the strartail would be generated only with the NIGHT images ...)

[EricClaeys]

@ova103,

If your daytime images are DARKER than your nighttime images you may want to adjust the daytime and/or nighttime "Mean Target" values. This is likely the cause of the problem you're seeing. The default daytime Mean Target is around 0.5 and the default nighttime is 0.2, which gives a darker nighttime image.
If you like your daytime images darker than nighttime images, you'll probably never get a good startrails as long as you're saving daytime images.
One of our to-do items is to allow users to specify begin and end times for images to be included in startrails, keograms, and timelapse. When/if we implement that it'll help you.

Did you try following the suggested steps at the startrails documentation page?
Try it with 20 or so nighttime images and keep adjusting the threshold until you get good results, then let me know what number worked.

allsky-config get_brightness_info basically runs:

grep Minimum: /var/log/allsky.log*

and displays the results along with average values. It suggests setting the Brightness Threshold to the "Maximum" value, or slightly below it.

Can you attach a typical daytime and typical nighttime image? Or a bunch of each images?

[ova103]

Thanks Eric.

The mean target has always been set at 0,5 for DAY and at 0,2 for night

I don't like my day time images darker than night. But if during the day it's grey and cloudy, and at night it's full moon and clear sky then day is darker than night I suppose.

I did the steps on the doc page for 10/7 and best treshold was 0.275

This is a startrail pic of 10/7 (it was cloudy the other nights )
threshold=0.275
startrails: Minimum: 0.179434 maximum: 0.782581 mean: 0.391017 median: 0.350245

some day pics of 10/7

night pics of 10/7

you can also see it on
https://www.vvscapella.be/OVA/html/allsky/index.php

I guess I need to put gain of Day higher than 0,5? and Maybe gain of NIght lower than 0,25?
Note: in mean time I changed mean target of day to 0,7. All the pics in this disucssion was mean target day = 0.5

[EricClaeys]

@ova103,
It doesn't hurt to change the daytime and nighttime "Target Mean" (not "gain" as you said). The higher the number, the brighter the image. You may want to have a smaller "Mean Threshold". I assume you are using the default Threshold of 0.1 ? Try 0.05.

0.1 Mean Threshold with a Mean Target of 0.2 means the actual Mean can vary from 0.1 to 0.3 (0.2 Target plus or minus the Threshold).

You can also play around with the "Stretch Amount" and "Stretch Midpoint" to make the stars appear brighter without changing the overall picture a lot. See the documentation page on Exposure, Gain, Stretch, and Saturation.

[jcauthen78]

My Observations & troubleshooting over the years:
I've had the same issue, wanting to capture both day & night images, but with the defaults, it makes the startrails a nightmare. My work-around solution was to save daytime images into a subfolder within the current date's folder (modifying the core saveImage.sh). Startrails doesn't dig through the subfolders, so it's been a small workaround that allows the startrails to only have access to the night-time images. The MiniTimelapse still works for the daytime images in the subfolder without modification.

But even with that workaround, the newer exposure algorithm makes the thresholding a challenge to dial in because most the images are fairly close to eachother in brightness regardless of stars or clouds.

I've debated working with some of the AI agents out there to find different methods to select which images go into the startrails production - be it starpoint/blob-detection, local contrast, edge detection, even frequency separation. All while maintaining the stability of the RPI computers, and using the existing data/metadata to not have to reprogram/design the entire wheel. Just need to find time.. mainly time to understand/interpret/intuit C++. Can't promise fast movement on the task though. just throwing out some random ideas that have run through my brain while trying to troubleshoot his pesky issue over the years

[EricClaeys]

@jcauthen78,
Until this thread I've never heard of anyone not being able to get a good startrails after adjusting the Brightness Threshold for their conditions. Having daytime images isn't a problem since they are brighter than the threshold so get filtered out. Any idea why your daytime images aren't filtered out?

Can you attach a typical daytime and nighttime image?

If Mini-Timelapse creation is enabled, saveImage.sh keeps a list of the last N images in allsky/tmp/mini-timelapse_files.txt, and when it creates a mini-timelapse, it uses that list. It doesn't matter where the images are stored.

What I plan to do at some point is allow the startrails and keogram commands to accept a file like 'mini-timelapse_files.txt" that contains a list of the images to put in the startrails or keogram. Then it would be easy in, for example, generateForDay.sh to create a list of all the nighttime images, and pass that list to startrails.
Limiting startrails to nighttime images may still be problematic depending on the Angle the user set. For example, if nighttime starts right after sunset, the first few dozen nighttime images will be pretty bright. I'm pretty sure that's why the startrails command uses a Brightness Threshold rather than simply look at nighttime images.
Something that could be causing problems is that the "Target Mean" values use the mean brightness of the center of the image, not the whole image, whereas the startrails command looks at the whole image, so if there's a lot of black border in the images the two mean values can't be compared.

[jcauthen78]

@EricClaeys , here are a few examples of my night time images. In order to get any type of startrails to be successful, I've had to narrow my Brightness Threshold down to 0.25. It still pulls in a lot of cloud images, making the image a wash of clouds and stars. I thought I'd brought it up a long while back - but honestly can't remember.

What I've noticed over the years: prior to the new exposure algorithm, the exposure time/gain was fairly constant at what it was manually set to - this allowed the user to tune the night time images decently. The non-startrails problem was that clouds and moon would end up blowing out the images at times, skyrocketing the brightness of the cloudy and moon filled images, hence the need for the new exposure. The skyrocketing is what made the thresholding work so well, clear images were usually always darker, so they could be pulled out easier. Cloudy usually always brighter, along with the moon.

With the new exposure, the cloudy images & some of the moonlit images are pulled down decently to not be as glaringly different. But this brought the image brightness's fairly close together regardless of clear/cloudy, making the threshold value a lot harder to set to have it be reliably successful from night to night, week to week, and over the year. If the star & the cloud images are the similar brightness, they're both making it into the startrails.

[EricClaeys]

@jcauthen78, Jim,
Are these images in 24-hour time, so the first one is 8:39 AM?

I understand what you're saying - improve the auto-exposure algorithm and it makes startrails correction harder.

@ova103, @jcauthen78,
take a look at this Issue on improving startrails. I think some of the ideas we're talking about will significantly help.
If you have other ideas, please chime in.

[ova103]

@jcauthen78, Jim, Are these images in 24-hour time, so the first one is 8:39 AM?

I understand what you're saying - improve the auto-exposure algorithm and it makes startrails correction harder.

@ova103, @jcauthen78, take a look at this Issue on improving startrails. I think some of the ideas we're talking about will significantly help. If you have other ideas, please chime in.

I did not fully undertand what Jcauthen is writing but I realised one thing: limiting the startrail program to night is what I believe we should do, but it's not the complete solution, because there will be at night cloudy images you'd like to avoid in the startail as well. But as they will be quite dark it's maybe not such a big issue after all. To get rid of those images I think it can only be done with a clear sky module, or a module that detect clouds ... Playing with brightness treshold to avoid cloudy night images seem to me impossible.

[jcauthen78]

@EricClaeys, The provided night images were from a single night to give an example of clear'ish, partly, and cloudy. The daytime image was this morning, a typical view of the daytime on my system. I can provide other examples if its easier.

I'm reading through the linked issue on improving startrails, but having trouble connecting the threads in my head (personal mental/focus issue, not content issue). Going to leave the fragmented thoughts here as to not cloud/junk up the other thread.

One thing I saw that would make life easier for typical users in that thread: Day & Night lables on the image names. My brute-force-workaround for this was to created a ./d/ folder for day images via the saveImage.sh. Having the file names clearly named day and night would be helpful for most - to some extent. Personally on my system though, it'd generate THOUSANDS of images in the same folder (especially on a cloudy night). I'm a non-typical user, as you know - my Day folder contains 3540 images for yesterday, and night (mostly cloudy) contains 1046 images. So, a singular folder containing 4,500+ images would be troublesome to navigate for the rare setup like mine - I'd probably brute-force it back to using a /day subfolder for daytime images (if no option was available). Why so many daytime images? i enjoy watching some of the larger storms develop and pass over - have caught some amazing storms with wall-clouds, roll-clouds, and even a developing funnel-cloud. again.. non-typical use for AllSky, but a bonus.

The --run-test option would be great, maybe taking the current threshold and stepping up and down from it by 0.01 or 0.02 5-10 steps each side? putting the threshold # in the file name itself, or overlaid on the image for easy visibility. The idea of saving some of the details in a ~/allsky/images/DATA/ folder would be helpful for some.

I thought there used to be a module for saving metadata to a file, logging the image data to a CSV/json file (name, exposure, gain, brightness, mean, starcount (if module used), etc...) - could have been something I was musing but never pushed forward on though. This may provide a quick way for Startrails to parse the file that already contains the data, instead of generating on the fly.... although the generation isn't that intensive on the fly from what I've noticed.

The AllSky AI module had some amazing potential - sadly it financially wasn't viable for the developer :( I was digging through some of the potential uses of separating out the cloudy vs clear (again, into a csv or json). He did a beautiful job with its development.

One thing I've also mused about recently is using the Starcount module - haven't had a chance to fully pull it apart and test it though. Currently it counts the stars within the masked area as the image is processed - I want to try and Log those numbers to a CSV/json (either within the current nights image folder, or the potential future images/DATA/ folder). Potentially use the starcount # as a threshold to generate an image list to feed into the StarTrails.. I haven't seen this be considered by others yet.. and it's just a rough idea for right now.. Maybe I'll tear the module open today..

[EricClaeys]

@jcauthen78,

having trouble connecting the threads in my head

The 7 items i the Issue list was a brain dump so in whatever order my brain decided. Possibly the only connection is they would all improve startrails.

---

Day & Night labels on the image names.

We most likely wouldn't change the names of the files to add any information. That would break all sorts of things that look for files with names of image-YYYYMMDDHHMMSS.jpg. It would also make for very long and unwieldy names. And putting images based on a criteria would work when there is only 1 criteria, e.g., DAY/NIGHT, but not when there are multiple criteria.
Instead of changing the file names, we would create an entry in the new database for every image, and that entry would contain all the information about the image we want. Allsky would query the database for things like "list all images for 20250731 taken at NIGHT with a MEAN value greater than 0.2 AND amount_of_clouds less than 5% AND number_of_airplanes = 0 AND ...", then pass that list to the startrails command to create the startrails.
If needed, we could probably write a web page to allow users to run a query without having to know how to write queries. For example, they could enter the values like 0.2, 5%, and 0 into a form and it would create and run the query.
Do you think this way would work?

a singular folder containing 4,500+ images would be troublesome to navigate for the rare setup like mine

I'm thinking with the database you won't need to navigate anything. I understand that "seeing" the files in a "day" or "night" folder is easy to comprehend visually, though

---

Re: --run-test (or a better name!)

putting the threshold # in the file name itself, or overlaid on the image for easy visibility.

Definitely in the file name - the names have to be unique, and hopefully also on the image, although maybe not in v2025.xx.xx since that will require changes to the module and overlay systems, and IMHO is a really nice thing to have, but still works without it.

stepping up and down from it by 0.01 or 0.02 5-10 steps each side?

It's easy to adjust the parameters and with only 50 or so images it's fairly quick to run so we could error on the side of "more" startrails to pick from rather than "fewer".
My biggest concern is having so many options that an advanced person would appreciate but that would overwhelm/confuse a beginner. I am fine if we decide to have an "advanced" mode that exposes a lot more options.

The idea of saving some of the details in a ~/allsky/images/DATA/ folder would be helpful for some.

What data are you thinking about?
Let's assume we use the database in v2025.xx.xx to hold all this data.

---

I thought there used to be a module for saving metadata to a file, logging the image data to a CSV/json file

@Alex-developer, can you please answer this paragraph?

logging the image data to a CSV/json file

Just thinking out loud...
For all the modules that produce per-image data like star count, we may want them to also (optionally) update the image's database entry. Then we'd have all the data needed for the database query to generate the list of images to create a startrails from.

---

Re: star count module saving data

I haven't seen this be considered by others yet.

I just added this as number 8 in the Issue.

[jcauthen78]

@EricClaeys ,
I appreciate the feedback & ideas. I'm mis-using the quote feature for replies, wasn't sure how to make the replies look clean 😅 still learning GitHub..

Day & Night labels on the image names.

I agree 100% - leaving the file names as-is is the way to go, helps continuity across the whole system.
I use SAMBA for quick file access - mainly grabbing a specific frame of interest I see in the timelapse (meteor, aurora, etc). Browsing them all in the WebUI hasn't ever been comfortable to me, especially when the folders have LARGE amts of images. SAMBA is my first install on a fresh system 🤓 so the /d folder is my way of sorting out the thousands of daytime images.

~/allsky/images/DATA/

Not sure what should or will go into it, but it makes sense to have some sort of storage for the json/csv stuff, or even the Database idea.

Database

I think this is a great way to track the metadata. Would it purge the entries as it purges the old nightly folders? My suggestion of JSON or CSV is just for brevity & ease of access by the ordinary user. Having it accessible via a page in the WebUI would work as well. Not sure exactly what data should go into the DB, but the basics for sure; image name, actual exposure, actual gain, image mean/brightness, starcount (maybe), clear/cloudy (if a good method is developed). Not sure how much, or what other should be logged. For my testing, I just grabbed it all for the JSON file.

Export Module

I found what I was thinking about, ALLSKY Export, but it only generates per-image data, not cumulative. So I cobbled together the rough metadata logger for my testing, dumping to a nightly JSON.

[EricClaeys]

@jcauthen78, I don't know how you did your quote feature (I'm still learning, too). What I do is start a line with a ">" then paste the part I want to quote after the ">". The key is that there needs to be a blank line after the quote (which can contain many lines), and before the reply. The "quote" icon (4th from the left) adds the ">" but I find it easier since my hand is already on the keyboard to just type the character.

Re: Day & Night labels on the image names.
We'll need to think of an easy way for you to "see" a list of files, like the daytime ones, without actually having to store them in separate directories. My gut tells me storing the images in different locations based on multiple criteria isn't going to work.
My gut also tells me that you aren't the only one that'll want to do this.

allsky-config samba is one of the first things I run on a new system, after downloading Allsky (it doesn't need to be installed to run it, just downloaded via "git clone...").

Re: ~/allsky/images/DATA/
I'm 99% certain that what you planned to store there is functionally the same as what would go in the database.

Re: database

Would it purge the entries as it purges the old nightly folders?

Alex and I talked about this but haven't decided anything. It's definitely one area where storing the info in allsky/images/YYYYMMDD would be easier, since the nightly purge removes the whole directory, so you'd get the metadata purge for free.
On thing we discussed is having a new setting in the WebUI to specify how long to keep the per-image data. It would likely go next to the "Days To Keep" setting since it's related. This allows for the possibility to keep the database data longer than the actual images, which might be useful to look at trends or something else (that we couldn't think of but at least my gut tells me might be useful). I think we'll need to purge data at some point or else the database will grow to large and fill up the disk. The size will be impacted by the next point below:

Not sure exactly what data should go into the DB

I'm not either! I'm thinking initially maybe everything that saveImage.sh gets passed to it (run grep ": saveImage.sh .*DATE=" /var/log/allsky.log | tail -1 to see that list). Then modules that create per-image data, like Star Count, would add their data.

Re: Export Module

but it only generates per-image data, not cumulative

What "cumulative" data?

[EricClaeys]

@ova103,

limiting the startrail program to night is what I believe we should do, but it's not the complete solution, because there will be at night cloudy images you'd like to avoid in the startail as well.

Agree 100%
Let me know if the Issue doesn't cover everything you are talking about.

there will be at night cloudy images you'd like to avoid in the startail as well. But as they will be quite dark it's maybe not such a big issue after all.

Clouds at a dark site are black but in a light polluted location they can be very bright. The dark clouds could easily be below the Brightness Threshold setting and hence added to the startrails.

---

Being "newer" and maybe "less experienced" than the rest of us I view as a positive which we need, since I want to make sure Allsky is usable for folks like you. I think that's our biggest user base.
If you see something we're considering doing that you think will confuse/frustrate people, please let us know so we can eliminate the confusion.

Thanks - Eric

[craigim]

I've been playing around with my own custom solution to star trails in python. I see the same issue where I have the gain dialed up for night to get stars through the light pollution here in Los Angeles, but that means my star trails are similar to the ones posted above no matter how I set the threshold.

The method I'm using (still really just a jupyter notebook at the moment) uses Skyfield and the lat/lon from the config file and calculates the times for astronomical twilight for that day. It then filters the file list using the timestamp in the filename so that only night images are included in the calculation. Those are then combined using pillow. I'm still working on the best way to do the threshold to get rid of extraneous cloud images. The last step is to use a mask and pillow's autocontrast to darken the sky and enhance the stars. Tonight's is below. That's a lot of stars to pick out of the sky 2 miles due east of LAX (the approach path is pretty obvious to the north.)

Rather than a database or json or trying to store metadata in the filename, have you considered populating the exif fields?

[craigim]

The way I would use the exif fields would be to normalize the brightness of each image by both extracted gain and exposure time to get counts/second at gain=1 when doing thresholding. Then it doesn't matter what your target brightness is or if you're using autogain or not.

[EricClaeys]

@craigim,
thanks for chiming in. Let us know how your tests go. If they work we should consider implementing it.

It then filters the file list using the timestamp in the filename so that only night images are included in the calculation

That'll come for free when we write per-image information the database since it'll include the NIGHT vs DAY information and won't require first calculating the astronomical twilight.

Those are then combined using pillow.

Can you explain more? I'm not following this (and don't know what "pillow" is, either!).

The last step is to use a mask and pillow's autocontrast to darken the sky and enhance the stars.

How does the autocontrast compare to using Allsky's "Stretch" settings?
I can see where using a slightly blurred mask would be nice so buildings, trees, and black borders (or whatever the user wants) don't get modified. The next version of Allsky will support masks in an easier fashion so it might be fairly easy to add a "Use mask when stretching" setting to the WebUI.

Rather than a database or json or trying to store metadata in the filename, have you considered populating the exif fields?

Yes, but it's too limiting (most of the fields don't have equivalent exif fields) and doesn't work with .png files (or possible other future formats). Plus, it makes querying for a set of attributes extremely slow since you have to look at every image. Doing a database query should be super fast. The capture program for RPi cameras used to write some basic EXIF data but we disable that part of the code for the reasons above.

The way I would use the exif fields would be to normalize the brightness of each image by both extracted gain and exposure time to get counts/second at gain=1 when doing thresholding. Then it doesn't matter what your target brightness is or if you're using autogain or not.

Would this be done when the image is saved, or after all images are saved and a startrail is ready to be created? And would it actually modify the saved image, or just record the normalized brightness and/or counts/second?
This data could be calculated using the database if it had the necessary information, correct?

---

Can I ask you to chime in on this Issue to help the team looking into improving startrails? And even join the team?

[craigim]

Pillow is an image processing library for python.

After I've selected my list of files, I loop through them. Before the loop, I create a blank image that is all black and the same size and my allsky images. Inside the loop, I use ImageChops.lighter method which compares images pixel-by-pixel and chooses the one that is "lighter". On the first iteration, no matter what image I have, it's automatically lighter than the black image. From there, stars are always lighter than the sky, so those pixels persist. Unfortunately, clouds in LA are also always bright, so those are rather intrusive. The example above is from a cloudless night.

My attempts at removing clouds has involved first creating a mask by making a numpy boolean image with True values in a disc centered in the middle of the frame and with a diameter 90% of the frame height. That matches my setup pretty well. I have tried a couple different strategies for distinguishing clouds, from just converting to greyscale and then thresholding to trying a normalized difference of (B-R) / (B+R) (clouds are white, so that index would be close to zero but light polluted sky is blue, so that value is large. That won't work for places with darker skies). The latter is promising, but the processing on my rpi5 goes from 7 frames processed per second to 1 frame every 2 seconds, so there is work to be done. However I do it, I stick an if statement into my loop and only combine a frame if it passes the threshold.

Finally, I use the mask and the ImageOps.autocontrast function and the mask to rescale the final image. It uses a histogram to do the stretching with an option to clip outliers, which I find to be more set and forget than the Allsky stretch method. In the posted image, if I don't autostretch, the background looks like daytime from the light pollution.

To your last question, I would not use the time and gain to correct the images themselves for display. My rpiHQ camera goes from as low as 2 microseconds exposure at noon to my maximum set of 20 seconds at night. That's 7 orders of magnitude of exposure which would require representing the brightness as a float64 to capture that dynamic range properly. However, when scanning through images to gauge brightness, you will have far more luck if you do image_brightness / gain / exposure_in_seconds. Then, it does not matter what your target brightness is because that will be normalized out by the exposure and gain. Your daytime frames will be much brighter by orders of magnitude. I bet you could even calculate the sky brightness and report out a bortle index, though that would take more knowledge than I have to do that calculation. You could indeed pull that from the database.

I will take a look at the issue and see if I have any additional comments.