Skip to content
Snippets Groups Projects

Draft: Exploring updating dataframe inplace

Closed Julia Signell requested to merge preprocess into preprocess-db

This branch uses an allpixels dataframe that the allfires objects update inplace. Right now when you do Fire_Forward on this branch the first step is to eagerly load the preprocessed pixels.

I also did some experiments with writing allpixels to disk. It works ok, but it is way faster to rehydrate if you separately save some metadata about the fires: t_st, t_ed.

There are two other fields that would be very handy to save off but they are per-fire/per-t. Those fields are ftype and hull (and potentially fline if you really want to not compute anything later on). Once I realized I needed them I started thinking that maybe I should be thinking about something other than csvs.

Incidentally there is a lot of IO logic around saving data at a particular time and across time, but a lot of it seems to rely on needing to hold multiple allfires objects at the same time.

I like that in this branch the allpixels object is referenced by all the allfires objects but it isn't copied around. This is different from how it works in preprocess where each fire object has its own dataframe.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply
Loading