SteemWorld News ~ Big Server-Update ahead!

in steemworld •  10 days ago

I have made many major changes/optimizations on the SteemWorld API in the recent week and I will transfer them to the main server in the coming days. The implementation process may lead to some hours of downtime for a few features on SteemWorld. Just in case there should occur issues for some time, you now know the reason for it.

As mentioned before, I have built my own solution for handling and saving blocks without the need of running a steemd instance on my server. Since I have very limited RAM and disk space available and no money for a high performance server, I was on a mission to find a better way for accessing and storing the block transactions and I am very happy with the results.

New Compression Technique

A regular steemd instance uses a 'block_log' file to store the raw blocks in an uncompressed way. Additionally it needs to keep the parsed operations in RAM (or shared memory file on disk, if you can't shit RAM) which is not possible for me with my available hardware (I would need to run a full node for the things that I want to implement in the coming months). I am thinking about storing also some of the 'virtual operations' on disk but that is a different topic and it might get a bit to technically for most of you to talk about this stuff here.

Steemd block_log : ~ 95 GB
My solution takes : ~ 11 GB

Dude, how did you compress a 95 GB file to 11 GB when even the best compression tools like 'xz -z -9' come to a resulting file size of ~ 34 GB?

I will explain it more in detail in one of my next posts. Compressing redundant field identifiers is one of the major reasons for such a small file size and of course I am not saving data that is only being required for consensus features. Nevertheless my data sets contain all 'non-virtual' operations. My parsing routine decompresses only the data that is currently being needed so that my server will never run out of space. Compression takes place from a cronjob in a separate process (because it takes a while). More details soon ;)

Cached API Requests

One 'experimental' optimization that will be activated in the coming days is to make use of 'Cached API Requests' for the users on SteemWorld. Instead of requesting most of the general data directly from the public nodes for each client every 30 seconds, my server will do these requests only once, cache the results and send the cached data to the client. I am not sure if my server is able to handle the huge amount of requests but I think it is possible. If this works as expected, there will be much less requests to the public nodes from my tool.

I am glad to see that Steemit is working on solutions for the upcoming problems regarding to memory usage and (even if I have built my own data service) I think that SBDS (Steem Blockchain Data Service) is a big step in the right direction!

Multiple SteemWorld Tabs

I have thought about how to minimize the number of server requests even more and came to the idea of using browser cookies for request caching. Some users like to open 5 or more SteemWorld tabs for different accounts at the same time and this of course multiplies the number of requests for refreshing the data by the number of open tabs. For these cases I want to store the result of each (main) request in a cookie so that the scripts in the other tabs receive the data from there instead of making new requests all the time. This will also get implemented in the coming week.

Thanks for your support!

Now back to work... 'Delegation History' coming soon! I wish you could see how many things happen in background daily and how much work it is to build and provide such a stable tool.

STEEM to the moon! We are living in exciting times, friends ;)
Much love and peace to all of you ~

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  Trending

Klasse Arbeit lieber Chiller !
Wir leben allerdings in sehr interessanten Zeiten ;)
Mach nur weiter so, bald kommt niemand mehr drumrum dein Tool zu benutzen!
LG
Jan

@steemchiller for witness!!! a lot of people would vote for you. the time has come. go for it!

·

Thanks @pawos! I've spent days to get steemd working and came to the conclusion that it sucks (sometimes). I don't like to run software on my server that comes with unforeseeable behaviour and sometimes stops working without informing me about it. The mission not to miss blocks is a stress factor I don't want to have in my life and the resources it currently takes would definitely require a new server. I'm waiting for a more stable version and a possibility to create kind of 'checkpoints' automatically so that I don't need to wait a day for the blockchain replay if steemd got stuck for some unknown reasons.

Great resource. I use it every day.

I'm curious if the recent @steemitblog announcement concerning Hivemind has any relevance to the direction you are taking with an external database.

https://busy.org/@steemitblog/update-communities-hivemind

·

Hivemind may be a step in the right direction but I don't think that it will change much for me and my databases. It seems to store a lot of community data in the 'custom_json' operation and this proves that there is much more possible with the steem blockchain than many people might have thought before.

Since I'm already indexing the data in my own way, I'm currently not in the need of using Hivemind, but it is indeed a very interesting 'project' for many developers out there. I'm curious to see where the journey will take us to :)

Sounds awesome! Even though I'm not that familiar with all the tech stuff;)
Thanks for your work for the community and your tool is always open in a tab, so I can check all ongoing activity right away.

·

Thanks! Yes, this one really contains much tech stuff. That's what I'm going through currently :) There is not much to show when most of my work is happening on the server side but I think my next post will be more about the front-end and I then can show a few nice screenshots again ;)

I wish you could see how many things happen in background daily and how much work it is to build and provide such a stable tool.

I have often thought this exact same thing!

I can run my laptop just fine, but when it comes to doing what you do, I have no damn clue :) So to me, you are a superstar and I totally appreciate that you do all of this to make everyone else's life easier on steemit! There should be awards for that. I'm always telling anyone who'll listen how awesome @steemworld is, and I've noticed that you always come around when you see your name mentioned and give me a sweet upvote. Seriously, it's something I never expect and I really do appreciate that. It shows me you are a real guy!

Cheers and thank you again for all you do!

First of all I love your work and want you to continue steemworld forever! I use it daily and without it being a steemian would suck.

I think your reward pool percentage calculation might be wrong. You should contact @dragosroua , he fixed it on his steem.supply page.
It could be the same or a similar calculation error.

·

Thanks! I'm thinking about removing the reward fund percentage completely because I realized that it can't be calculated based on a value from which the reward has already been subtracted. Maybe it would be better to show a percentage for how much of the total pending rewards goes to the user.

The calculation of a correct value is a bit more difficult than one might think, because of the SBD payout. All rewards come in SBD (worth of 1 USD) but the totals in the blockchain are measured in STEEM. I will think about all this when I find time in the coming weeks... :)

·
·

I know, it is really complicated. I could not do it if I tried...

Thank you for what you're doing with SteemWorld. Great tool and I love it. Keep up the good work! Upvoted!

Nice work man! its very impressive all of what you're doing in the back-end. Even though I'm quite new to the space, I have already been using your tool. Thanks for your that contribution and all of the other tools and knowledge that I'll be able to engage with through your page. Already following you. Hope we can keep in touch, as I am interested in learning from all of your great content.
Cheers!

@DeXvid

Thank you for providing such a details analysis especially the pie chart on SteemWorld[dot]org. I have been using it daily. A witness vote as my appreciation for your hard work. Cheers.

Bravo.png   for this wonderful tool! SteemWorld is fantastic, especially since you got the delegations-in and delegations-out section working! I love the filters for account operations, the "VP ~> 100%" calculation, the "mention" history, and so many other things! This is one of the best toolboxes for the Steem blockchain, and your hard work is DEFINITELY appreciated! THANK YOU! 💙

Lieber @steemchiller,

kann es sein, dass die Mentions nicht mehr funktionieren?

Lieben Gruß, double-u

·

Danke für den Hinweis @double-u! Ich habe die Ursache gefunden und wie es aussieht, muss ich doch die Texte manuell nach @ durchsuchen, da in z.B. auf busy.org erfassten Kommentaren das Feld "users" nicht mit den verknüpften Usern gefüllt wird. Eigentlich ist das Feld dafür vorgesehen, damit der Zugriff auf diese Information schneller ablaufen kann und nicht jeder Entwickler eine Routine für die Ermittlung der Namen aus dem reinen Text erstellen muss...

Auf Steemit funktioniert das, daher wirst du meine Mention mit Sicherheit auch erhalten. Bin gerade etwas ratlos, aber ich denke, es führt für mich kein Weg daran vorbei, das auch manuell zu ermitteln. Ich werde das aber in meinem nächsten Post mal ansprechen, da ich finde, dass die verschiedenen Frontends wenigstens die Daten einheitlich erfassen sollten.

LG, Chiller

·
·

Danke für deine so schnelle Reaktion, lieber Chiller!

Ja, die Erwähnung von dir von eben wurde mir gemeldet.

Schönes Wochenende!

Und lieben Gruß, double-u

Amigo, te comento que yo lo uso a diario de hecho es uno de los mejores. y tu sabes que existen muchos. pero yo particularmente uso el tuyo. gracias por eso. Saludos desde Venezuela.

Friend, I tell you that I use it daily is in fact one of the best. and you know there are many. but I particularly use yours. thanks for that. Greetings from Venezuela.

Solicito tu permiso para hacer la traduccion de toda esta publicacion. asi llega a muchos Venezolanos que la utilizamos.

I request your permission to do the translation of all this publication. So it comes to many Venezuelans who use it.

·

Thank you! Feel free to make translations ;)

·
·

Thank you, I was waiting for your response. Tomorrow, Sunday, I'll add it up. I am sincere I am addicted to your program, thanks to him, I saw your answer.
I have a question when one uses the manual vote the connection uses the steemconet?

·
·
·

The manual vote feature uses the SteemJS API. It does not work with the Steemit Password. People should always use the 'Private Posting Key' for things like posting, commenting and voting. Since the keys are not being stored in a cookie or so, one must re-enter them after switching to an other account or closing the SteemWorld tab.

·
·
·
·

All SBD earnings from this article go to @steemchiller to fund the SteemWorld server

Noticias de SteemWorld ¡Big Server-Actualización a continuación!

·
·
·
·
·

Thank you very much! As I've seen, you don't earn much with your posts, so please take the SBD for yourself my friend ;)

·
·
·
·
·
·

Thanks friend. I really would like to help.
I propose only if you allow it. Make a definitive agreement to carry out the translation into Spanish of your publications.

I will comment, since I was a child I was taught to be a man of my word. and for me, the word is law. count on my humble help
I hope to continue growing.

·
·
·
·
·
·
·

It's okay. You should earn something for your translation, so please use the SBD from your post to grow your account :)

I read comments by others saying that yours is the best tool for steemit. I'll have to take their words for it, as I do not use any others. Why should I bother, when I have one that does everything I need so well?

Equally important is the way you always respond to questions, however silly or ignorant. I admire that you are able to visualise your way through the complex programming, while remaining such a warm and kind human being.

I just love your tool, and keep plugging it to the new users, it is so useful, a necessity in fact for a smooth and fun steemit experience.

Thank you!! gorgeous SteemWorld!!!

Thank-you so much for offering this pleasant and helpful website @steemchiller 💖👍💖

jgf.1602.jpg image by: @dandedign86

Awesome work! Such advancements are definitely necessary for future tools as well as your own.

Awesome! Although I don't know much about the techie stuff, but able to compress a huge 95GB to become 11GB is super cool! Keep up the awesome work! Congrats! And thanks!

Awesome work @steemchiller ! I use steemworld just as much as I use Steemit haha. I'm excited for the update, and I'm glad I saw your post in case there's some downtime. Thanks for all that you do!

@steemchiller - Million thanks for keeping update & upgrade your services.

But my big warning to the new user who never experience SteemWorld please stay away from it or otherwise you will be addicted like me. :-)

Cheers / Andy

·

otherwise you will be addicted like me

Thank you @andyjim! Yeah, that's true! I'm addicted too... ^^

·
·

Great and impressive work, congratulations.

So, you are completely steemd-less? Somewhat blasphemous concept :)

Yet I applaud you. These modern blockchain platform don't think about the space (and RAM) needed at all.

@steemchiller... Thank you for providing us with this really super useful tool.

Are you a witness @steemchiller ?

If not maybe you could be? if you are maybe a banner at the bottom of the page?

100% upvote from me, I use your service a lot thanks!

Some users like to open 5 or more SteemWorld tabs for different accounts at the same time and this of course multiplies the number of requests for refreshing the data by the number of open tabs.

Hmm - I think these "outrageous guys" should actively support your work. Go ahead!

·

Hehehe... ^^ Ich musste dabei auch an dich denken. Habe extra 5 geschrieben, damit es sich noch 'dramatischer' anhört. Vielen Dank übrigens für deinen Post! Kann den nicht teilen, weil es gerade nicht in meinen 'Masterplan' passt, aber du kannst die Einnahmen daraus für dich behalten. Sollst ja schließlich auch etwas mit deiner Arbeit verdienen ;)

Freitag der 13. war übrigens schon immer mein Glückstag, da ich die Gedanken-Matrix mal -1 multipliziert habe für positivere Ergebnisse und so... :)

LG, Chiller

·
·

Ich musste dabei auch an dich denken. Habe extra 5 geschrieben, damit es sich noch 'dramatischer' anhört.

5 - 7 wäre natürlich inyourface gewesen ;)

aber du kannst die Einnahmen daraus für dich behalten.

Das würde mir einfallen. Selbst wenn es 0.001 SBD wären, bekommst du die ganz akkurat (wie von einer Behörde) aufs "Konto" überwiesen :)

Hauptgrund für den Post war aber auch primär gar nicht da Megasummen zusammen zu bekommen, sondern einfach mal auf meine Art und Weise ein paar Leute wieder (oder überhaupt) mal daran zu erinnern, was du da Tag für Tag leistest und dass das auf keinen Fall eine Selbstverständlichkeit ist. "Good Guy - Bad Guy" - Chriddi hat den ersten Teil übernommen. ;)

Und wie meine Lieblingslehrerin hier auf Steemit immer so schön sagt: "Solche Posts sind wie ein großes Sozialexperiment".

Ich weiß wer mich hier monitort, weil ich öfter mal gewissen Kreisen auf die Füsse trete und aus den Reaktionen kannst du soooo viel heraus lesen. Besonders die NICHT-Reaktionen sind oft aufschlussreicher, wie eine Reaktion, wenn du weißt an wen der "Prüfungsbogen" verteilt wurde, aber Einige denken man merkt es nicht, wenn man den schnell in den Papierkorb wirft und so tut, als wäre man nicht anwesend gewesen :D

Freitag der 13. war übrigens schon immer mein Glückstag, da ich die Gedanken-Matrix mal -1 multipliziert habe für positivere Ergebnisse und so

Sehr cool - hier wärst du damit sowieso auf der sicheren Seite. Hier ist DIENSTAG der 13. der "Pechtag", wobei ein Freitag mit diesem Tagesdatum positive, zukünftige Ereignisse zur Folge haben soll.

Andere Länder, andere Sitten.
Man muss sich nur immer die Positiven raus picken.

Gruß über den großen Teich in die europäische Nacht!

steemworld is a great tool that has helped the growth and understanding of the payments and statistics of my account in steemit.

Ich finde es faszinierend einen Einblick in diese Welt zu bekommen. Da merke ich erst, wie ignorant ich bzgl. sowas bin. Ich wüsste nicht mal wie man das nennt. Softwareentwickler? Programmier? Wie nennt man diese Arbeit?

·

Es gibt einen Unterschied zwischen Entwicklern und Programmierern, aber in kleinen bis mittelständischen Unternehmen (in denen man meist eh Arsch für alles ist) wird die Entwicklung, Programmierung und Wartung oft von nur einer Person erledigt. Ein Entwickler muss nicht unbedingt auch der Programmierer einer Software sein. Er könnte die Logik vorher schriftlich und bildlich erledigt haben und die Aufgabe der Realisierung an einen Programmierer weitergeben. In größeren Unternehmen sind diese Abläufe natürlich deutlich besser strukturiert.

Da ich ja alles alleine mache, bin ich gerade Entwickler, Programmierer, Tester, Webdesigner und Vertriebler in einer Person. Viel Zeit für andere Dinge bleibt da natürlich nicht mehr, aber ich hoffe, dass es sich eines Tages gelohnt haben wird.

Not only the best app for knowing what I'm doing (or someone else) on the STEEM ecosystem, but also constantly updating and improving to stay that way. And now, trying to get even faster. This is great.

Did I read somewhere that you might be or maybe have already added notifications to steemworld? (I haven't noticed a tab for it, but I could be blind).

Also, I thought you were a witness at one time. You should be! Lots of work, though, and you've got plenty of that going on already.

Thanks for another update and for all the coming improvements.

·

The desktop notifications are working. Perhaps you didn't turn them on last time you opened SteemWorld. You may have to clear cookies and load it again. I know I'm in the habit of closing those without allowing notifications, so you may have turned them off without out of habit.

  • They work on my Linux desktop and Windows desktop.

I agree it would be great to have another good witness to vote for!!!

·
·

That's where I saw it! It was at the top for a while as you logged in. I think I probably ignored it. So, there's no way to turn them on once you're in the app? I'm looking now but nothing leaps out at me. :)

So, I need to clear cookies first, then log back in? What if I have multiple pages up? Will I get beeped every time the other account gets something too?

·
·
·

I'm not exactly sure how to get it back.

  • You might try closing all your browser windows and restarting your browser. Go to SteemWorld and see if it gives you the option for notifications.
    • If that doesn't work, I'm guessing clearing your cookies should do it...
  • Good luck. Let us know what works for you...

Yes, I occasionally have other people's SteemWorld pages up and get notices from them...

·
·
·

In case you've just ignored the hint and you didn't click on 'no' (not allow notifications) it should show up again. The option 'Desktop Notifications' in the settings on SteemWorld should not be set to 'Disabled' of course. Depending on the browser you are using, there should be a section for notifications in your browser settings where you can configure the behaviour for each website.

·
·
·
·

So, I found the notification settings for my browser, and I think I've found the desktop notifications under settings. The browser is now set to accept and the site settings is set to show all. Thanks for your help.

Herzlichen Dank für diese tolle, einfache und sehr informationsreiche Applikation und vor allem auch für deine Arbeit. SteemWord ist einfach genial! Sonnige Grüsse

its a great news for the user and they get more and faster upload their browser, as it technical issue and i have limited knowledge about it. but it is very helpful for me since i have been used it for the first time. thanks for your contribution.

Great great - many many thx. SteemWorld is always running next my acitvities. For me currently the most useful tool here!

Please allow me to translate these papers into Indonesian language. Because this is very useful and keep in mind my steemian friends in my place. I keep using copyright @steemchiller. Thank you for your answer. Have a nice day, Sir @ steemchiller.

Awesome @steemchiller
I love SteemWorld, it has come my default steem homepage for me :D

thank you @steemchiller . I have run it and wrote the tutorial on my steemit account, based on my own experience so others can run it. It's a smart and very helpful feature. thanks.

Mein 100% Upvote ist zwar nicht viel.
Hoffe es läppert sich trotzdem ein bisschen was zusammen ;)

Wollte auch danke sagen für dein Tool. Das macht das Tägliche Steem Leben um so vieles einfacher. Vielen Dank für die Arbeit die du dir für uns machst.

You're doing an amazing job, we're so lucky to have you!!

Oh, man I don't miss the days of code & API development... So much has changed it makes my head swim. Keep up the fine work. We certainly appreciate it!

Regarding this steemword I want to campaign it in Aceh, Indonesia. The last post describes it. Apparently Stemian there welcomed it and responded very positively. And this feature is considered very complete and facilitate the users. Thank you @steemchiller

Good work..I appreciate your update news.
Thanks for sharing Valuable news.
I love steemworld.
Upvote resteemit done

Is there much of a difference in your site as compared to the "steemnow" site?

It's great that the project is going to be alive and we are going in the right direction

You just planted 0.29 tree(s)!


Thanks to @vladimir-simovic

We have planted already 4395.84 trees
out of 1,000,000


Let's save and restore Abongphen Highland Forest
in Cameroonian village Kedjom-Keku!
Plant trees with @treeplanter and get paid for it!
My Steem Power = 18674.05
Thanks a lot!
@martin.mikes coordinator of @kedjom-keku
treeplantermessage_ok.png

I like this steem
I appreciate your post.

This great news,I appreciate steemworld.

Will this update require some downtime? Or this will be done without effecting steemworld.org?