Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saving: Added Region Save Mode #60

Merged
2 commits merged into from
Apr 14, 2013
Merged

Saving: Added Region Save Mode #60

2 commits merged into from
Apr 14, 2013

Conversation

Nebual
Copy link
Collaborator

@Nebual Nebual commented Apr 14, 2013

A very scalable system. Sectors (8x8x8 blocks) are stored 4x4x4 in headerless
files, 2 bytes per block. Position data is not stored, but inferred based on
the byte offset within the file.

Whole regions (32x32x32 blocks) are read at once when world._show_sector asks.
Sectors can be written to individually, though without a dirty_sectors system (TODO)
it currently writes all sectors in memory to disk during a save.

Nebual added 2 commits April 14, 2013 01:39
This means you can build as high or low as you like, vertical travel is
equivalent to horizontal travel performance wise.

Also optimized world.change_sectors
A very scalable system. Sectors (8x8x8 blocks) are stored 4x4x4 in headerless
files, 2 bytes per block. Position data is not stored, but inferred based on
the byte offset within the file.

Whole regions (32x32x32 blocks) are read at once when world._show_sector asks.
Sectors can be written to individually, though without a dirty_sectors system,
it currently writes all sectors in memory to disk during a save.
ghost pushed a commit that referenced this pull request Apr 14, 2013
Saving: Added Region Save Mode
@ghost ghost merged commit 08ce2b2 into boskee:master Apr 14, 2013
@ghost
Copy link

ghost commented Apr 14, 2013

Saving is still very slow...

@BertrandBordage
Copy link
Collaborator

@avelanarius: I started working on #40 today. For the moment, creating a new world with this future SQL solution is slow (~ 30% slower than current creation), but any other save/load takes less than 0.5s. And it's done with SQLite on the same process than the game. When it'll be on another process, it'll be much faster. And if we still have performance issue, we can consider using PostgreSQL, which should solve anything.

Every SQL save takes more than twice the size of @Nebual's current system. Compressing with 7z divides it by 6. A compressed 100-sized world therefore takes less than 500 kB. So size will never be a problem. A really big world will never reach 250 MB if compressed.

However, it's still very experimental and we need @Nebual's solution.

@Nebual
Copy link
Collaborator Author

Nebual commented Apr 14, 2013

I believe loading 64 sectors (1 whole region file, or 32768 blocks) takes between 0.0005s and 0.004s under my Region system.

Theres a few odd behaviors with the Queuing system that I believe are slowing down sector loading considerably. And I notice that if you keep the urgent queue full (by constantly flying and thus loading new chunks) the hide_sector calls don't occur until its totally empty. I also don't think the world.delete_opposite_task aspect is working, though its a good idea.

@Nebual Nebual mentioned this pull request Apr 14, 2013
@BertrandBordage
Copy link
Collaborator

I have extreme lag since this pull request. Less than 40 FPS when I'm moving (sometimes even 15 FPS), the game freezes every second and there are dozens of bugs, including a ground in the sky, some crashes with KeyError, falling from every unloaded sector, screen update glitches, exposed blocks not calculated properly, etc. And sector updates are slower than a snail, even when the files were already generated in a previous game.

However, I must admin that opening an already created world is fast.

@BertrandBordage
Copy link
Collaborator

I also see an increase in RAM usage. I expected the opposite :(

@Nebual
Copy link
Collaborator Author

Nebual commented Apr 14, 2013

Odd, thats significantly worse than me. I'd be at 300fps standing still, dropping down to 120fps while loading chunks, the queueing prevented much framedrop I'm guessing. This is in pure python mode.

The "ground in the sky" is because my simple terrain generator is just blindly creating a layer of dirt at the bottom of all new Regions, including those in the sky. "exposed blocks not calculated properly"? "screen update glitches"?

Increase in RAM is both because more of the world is loaded now than before, and I'm not currently unloading sectors, so RAM usage will climb as you explore, but since theres nothing to explore and it resets when you reopen the game, I wasn't going to delay releasing this for that.

Edit: Also, you can switch back to the previous save if you need to.

@BertrandBordage
Copy link
Collaborator

That huge FPS drop I got is measured without Cython on an ASUS G74SX. So this is definitely not my computer that has a problem, though it's per-core power is limited.

You can also randomly see blocks that shouldn't be exposed: stone and ore blocks. In some games, I could see lots of "ore walls" under the ground, while waiting for sector updates. This is increasing the number of polygons to show, which decreases performance on computers with less GPU power.

The screen update glitches I saw (only a tiny square of the screen is refreshed) are maybe due to the chaotic freezes. I never got such glitches before, and no other OpenGL application was running at the same time.

@ronmurphy
Copy link
Collaborator

@BertrandBordage Quote:"You can also randomly see blocks that shouldn't be exposed: stone and ore blocks. In some games, I could see lots of "ore walls" under the ground, while waiting for sector updates. "

I saw this also. If you remove an Invisible block, then it, and it's neighbors, show up. Interesting.

@BertrandBordage
Copy link
Collaborator

I also note that the lazy queue, which is responsible for hiding sectors, is also extremely slow. This really shouldn't be.

@BertrandBordage
Copy link
Collaborator

@ronmurphy: It's normal that the "neighbours" blocks show up when you remove or place a block. It's thanks to World.check_neighbours. It's a good way to only load in your graphic card the blocks that should be visible. Minecraft works the same.

@ronmurphy
Copy link
Collaborator

@bert I am trying to do something useful for once, I am trying to make a new sub from the existing World.initilize, at the land generation, that does +16 to current position to generate new terrain.

Not to sure how well it will work, or even if it will, but i will post results once i get it working.

@BertrandBordage
Copy link
Collaborator

"+16 to current position"?

@ronmurphy
Copy link
Collaborator

yes, up to 16 blocks away from current position should be generated, not more ... this could be changed, but i think that it will keep terrain generation time down, say, from +32 to current position.

@BertrandBordage
Copy link
Collaborator

OK. This is exactly what I'm currently doing in my SQL experiment ^^

@ronmurphy
Copy link
Collaborator

ok :D I had to stop as how i did so much not-working code i had to do a revert...
Just more to learn more python!

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants