-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix sqlite migration #3
base: master
Are you sure you want to change the base?
Conversation
Hey guys, first of all thanks for the repo and thanks for this PR. I'm trying to run the script through a Docker container as I don't anything Python related directly on my machine, it's not a language I use/know personally. I've created the following
But when it runs the
I have no idea what to do with this 😓 Could anyone help me out to run the migration code? Thanks a lot! Oh and the build command I'm using: |
A bit busy with work right now, but in your docker file you're pulling the latest home-assistant core. This PR is up to date with release 2022.6.7 You can try to instead pull that version using This script would need to be verified to work with the latest home-assistant core release. |
The issue is morelikely that he tries to use an old python version. @maxime1992 try to use the python image (untested):
|
Hey guys, I've managed early this morning to get it working 🙏 It worked like 90% through, I think I ran out of RAM after that but I'll just kill a few things and re-run it it should be perfect 👌. When I get some time I'll explain how I've done it with Docker, which should help other people trying to run this script :) Thanks for your answers and both of your work on this! |
Ok so I can run the script, but while running it I'm getting errors like this:
Any idea what I should do to fix this? |
I can provide a fix tomorrow. |
Wow that's fantastic @fabm3n thank you so much! 🙏 I forgot to post it but there's also this in the error logs:
I've used dbeaver to look at the database and the different attributes. I don't think they've changed so yup I think I'm lost here... Thank you so much :) |
I will send you a quick mail. Can you send me your database so i can try it on my own? |
Can you check why these entities have no attributes? |
@fabm3n can you give some guidance on what to do please ? |
Ok not sure of how helpful that info is going to be... But I've ran this query: I've taken a look into what it might be, and I think they were mostly sensor that:
So essentially... I'm pretty sure I don't care about these 600 values, and I suspect your fix might help already. I'll try to have a go! |
Ok I tried with all your latest updates, here's the error I got:
|
Victory!!! It's all working I've got all the data in Influx! Thanks to both of you for the help and patience 🙏 ❤️ ! I'll try to explain (soon) what I've done with Docker as it helped to avoid some Python issue, in case someone else needs it. |
@Maaxion I reckon this is good for merging and it's been tested for a migration while I was on HA 2023.1.4 FYI :) |
As promised, in case anyone else wants to run this with Docker:
Note: If this PR is not merged yet, make sure to apply the changes first (or instead of cloning from Maaxion's repo do this:
Notes on the above:
I had issues as explained in the comments above while trying to run a Docker container with Python myself, hence the build of HA first and then start of that image to put the code of |
Hello @fabm3n @maxime1992 , Traceback (most recent call last): any idea ? |
Hey, I'm no Python expert (actually never wrote any) but look at the few last commits made by @fabm3n. It's probably a matter of doing just the same for your field in question. |
On a separate note, I started to use Kibana and noticed that data prior the migration and new data coming straight out of HA have a difference: Unsure why it's not considered the same 🤔 Anyone has an idea of what to do here? Fixing the data in influx would be ideal, but if it's too much troubles I can just re-export again the entire database as I've kept it running in // on HA in case I had any issue. EDIT: In case anyone else comes across this, I've managed to fix my grafana query by adding this at the end of my query: |
That's good! So ignoring in your case is not a problem. |
Do you still have your "old" database and can check which data has a friendly name set as null? Like you did it for the status. |
Hi @max5962, |
I didn't went deep into my data so i never had such a issue. That's why i don't try to fix this :) |
I have, I may have a look at some point but not right now
Oh yeah it's fine, as per my edit in the message you quoted, the only fix needed is to remove that column. And a simple fix in this script would be to drop the line that adds the "source" and sets it to HA. |
I succeed last week to retreived my data by changing the request by : With previous database format, the time_fired was always NULL, so i converted last_updated_ts, to the desired format. I just want to use my history data in an Grafana, so i don't care if its not totaly exact ( cf "||.33195 ).
|
Hi everyone, |
Hi @xaviergriffon would you be so kind and show how you changed your code? |
Hi I need some help, the python seems to execute but the Sqlite data entity_id is failing some validation ?? Any ideas what happening here, not very skilled at Python (.venv) pi@Pi4:~/homeassistant2influxdb $ python3.11 homeassistant2influxdb.py --type SQLite --database home-assistant_v2.db Is the code trying to write the Influx database as this in docker ( Not sure how the code knows the database name to access an existing Influxdb whcih has some data already loaded ??? (.venv) pi@Pi4:~/homeassistant2influxdb $ docker exec -it influxdb influx
W,domain=sensor,entity_id=myenergi_eddi_23468676_internal_load_ct1 Any help appreciate ... |
I'm seeing this same issue. Were you able to resolve it? |
Sorry no updates ...I have been on holiday I will have another look in the next few weeks... I'm guess some data in not in the correct format |
@waltonbp Hi, having the same issue. Have you found a solution? |
I'll just re-share the message I posted a while ago: #3 (comment) I think it may have been overlooked and could help a few. I'm really not an expert in that area and it took me quite a few hours to come up with that, so I'm afraid I won't be able to help much further but as soon as I succeeded, I wrote that message while everything was still fresh in my mind so you may want to have a look |
Hey all! Any solution to this? I have issues with dependencies not working, which then means the script won't run. |
Have you checked my comment above ? It's not as simple as one may like but I'm pretty sure that'd still work |
I resolved all the dependencies issues, and got the script mostly running, but now it chokes when it hits a specific line of code. Something about the sql query not being a valid one. Would running it in Docker help that? I'm trying to import from a SQLite database file into InfluxDB using their v2 API |
found some bugs while migrating my sqlite database to influx db