Skip to content

Commit

Permalink
deploy: a69334f
Browse files Browse the repository at this point in the history
  • Loading branch information
@ewinnington
ewinnington committed Mar 20, 2024
1 parent 947a14c commit a4a9dcf
Show file tree
Hide file tree
Showing 157 changed files with 3,270 additions and 2,328 deletions.
2 changes: 1 addition & 1 deletion about.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ <h2>Feeds</h2>
<section>
</section>
<ul class="copyright">
<li>Copyright © 2023</li>
<li>Copyright © 2024</li>
<li>Design: <a href="http://html5up.net">HTML5 UP</a></li>
<li><a href="https://wyam.io">Generated by Wyam</a></li>
</ul>
Expand Down
158 changes: 81 additions & 77 deletions feed.atom
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,88 @@
<id>http://ewinnington.github.io/</id>
<title>Eric Winnington</title>
<link rel="self" href="http://ewinnington.github.io/" />
<rights>2023</rights>
<updated>2023-07-24T14:48:14Z</updated>
<rights>2024</rights>
<updated>2024-03-20T12:54:01Z</updated>
<subtitle>A collection of thoughts, code and snippets.</subtitle>
<entry>
<id>http://ewinnington.github.io/posts/HttpClientCompression</id>
<title>Receiving compressed data from an http(s) endpoint</title>
<link href="http://ewinnington.github.io/posts/HttpClientCompression" />
<updated>2024-03-20T11:00:00Z</updated>
<content>&lt;p&gt;With the amount of data that we are passing around across services, it is often beneficial to use compression on the data to reduce the transmission time. Modern platforms and algorithms are now very efficient at compressing regular data, particularly if that data is text or json data. &lt;/p&gt;
&lt;p&gt;If the developer of the endpoint has prepared their service for compression, the client must still indicate that they are ready to receive the compressed data. Luckily, most implementations of modern http clients in R, Python, JavaScript and Dotnet support compression / decompression and are seamless for the client. This means that you can set the compression headers on and simply benefit from compressed data being received. &lt;/p&gt;
&lt;p&gt;We can also check in the Content-Encoding header which compression was used. I've found that example.com is sending responses compressed with gzip.&lt;/p&gt;
&lt;h2 id="python"&gt;Python&lt;/h2&gt;
&lt;pre&gt;&lt;code class="language-python"&gt;import requests

url = &amp;quot;http://example.com&amp;quot; # Replace with the actual URL you want to request

# Specify the accepted encoding methods in the headers
headers = {
'Accept-Encoding': 'gzip, br',
}

response = requests.get(url, headers=headers)
print(response.text)

# In case you want to see if it was compressed, you can check via the headers
#if 'Content-Encoding' in response.headers:
# print(f&amp;quot;Response was compressed using: {response.headers['Content-Encoding']}&amp;quot;)
#else:
# print(&amp;quot;Response was not compressed.&amp;quot;)
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="r"&gt;R&lt;/h2&gt;
&lt;pre&gt;&lt;code class="language-R"&gt;library(httr)

# The URL to which you're sending the request
url &amp;lt;- &amp;quot;http://example.com&amp;quot;

# Setting the Accept-Encoding header
response &amp;lt;- GET(url, add_headers(`Accept-Encoding` = 'gzip, br'))

# The content of the response will be automatically decompressed by httr, so you can access it directly.
content(response, &amp;quot;text&amp;quot;)
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="c"&gt;C#&lt;/h2&gt;
&lt;p&gt;In C#, for some ungodly strange reason, the standard HTTP endpoint doesn't decompress for you automatically unless you add a decompression handler - see handler &lt;a href="https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.automaticdecompression?view=net-8.0#system-net-http-httpclienthandler-automaticdecompression"&gt;HttpClientHandler.AutomaticDecompression&lt;/a&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-Csharp"&gt;using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using System.Text;

class Program
{
static async Task Main(string[] args)
{
HttpClientHandler handler = new HttpClientHandler();
handler.AutomaticDecompression = System.Net.DecompressionMethods.GZip; //Adding automatic Decompression means that the accept headers are added automatically

using (var client = new HttpClient(handler))
{
string url = &amp;quot;http://example.com&amp;quot;;
HttpResponseMessage response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();

Console.WriteLine(await response.Content.ReadAsStringAsync());
}
}
}
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="javascript"&gt;JavaScript&lt;/h2&gt;
&lt;p&gt;it is so easy that you don't even need to do anything else than setting the gzip: true for the support&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-JS"&gt;const request = require('request');
const requestOptions = {
url: 'http://example.com',
gzip: true, // This is all that is required
};
request(requestOptions, (error, response, body) =&amp;gt; {
// Handle the response here
});
&lt;/code&gt;&lt;/pre&gt;
</content>
<summary>&lt;p&gt;With the amount of data that we are passing around across services, it is often beneficial to use compression on the data to reduce the transmission time. Modern platforms and algorithms are now very efficient at compressing regular data, particularly if that data is text or json data.&amp;nbsp;&lt;/p&gt;</summary>
</entry>
<entry>
<id>http://ewinnington.github.io/posts/devcontainers</id>
<title>DevContainers - The future of developer environments</title>
Expand Down Expand Up @@ -1528,79 +1607,4 @@ I'm using the &lt;a href="https://www.npgsql.org/index.html"&gt;Npgsql drivers&l
</content>
<summary>&lt;p&gt;In the context of Docker and Jupyter Notebook, it's interesting to note that there exists a Nuget that allows C# to control docker. So, yes, it is possible to launch a Postgresql database, on docker, inside a Jupyter notebook!&lt;/p&gt;</summary>
</entry>
<entry>
<id>http://ewinnington.github.io/posts/jupyter-sqlite-csharp</id>
<title>Testing SQLite in C# Jupyter notebook</title>
<link href="http://ewinnington.github.io/posts/jupyter-sqlite-csharp" />
<updated>2019-11-12T20:00:00Z</updated>
<content>&lt;p&gt;Now that we have &lt;a href="/posts/jupyter-notebook-csharp-r"&gt;Jupyter Notebooks with C# installed&lt;/a&gt;, using it as an environment to play with SQLite is very easy. &lt;a href="https://www.sqlite.org/index.html"&gt;SQLite&lt;/a&gt; is a relational database that is small in footprint and self-contained. It also has a great in-memory mode which is perfect for playing around in a Jupyter notebook.&lt;/p&gt;
&lt;p&gt;You can access my &lt;a href="https://github.com/ewinnington/noteb/blob/master/SqliteInteraction.ipynb"&gt;SQLite example notebook here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;You can now launch this online and follow along:
&lt;a href="https://mybinder.org/v2/gh/ewinnington/noteb/master?filepath=SqliteInteraction.ipynb"&gt;&lt;img src="https://mybinder.org/badge_logo.svg" class="img-fluid" alt="Binder" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;You can create a C# Notebook from the file menu of Jupyter.&lt;/p&gt;
&lt;p&gt;&lt;img src="/posts/images/jupyter-notebook-csharp-r/new-notebook.png" class="img-fluid" alt="new-notebook" /&gt;&lt;/p&gt;
&lt;p&gt;We need to pull in the nuget package &lt;code&gt;System.Data.SQLite&lt;/code&gt; to interact with the database.&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-CSharp"&gt;#r &amp;quot;nuget:System.Data.SQLite&amp;quot;
using System.Data.SQLite;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;#r&lt;/code&gt; is used to reference a dll or a nuget package. If you prefix the command with &amp;quot;nuget:&amp;quot; then the jupyter notebook will download the nuget and add it as a reference. Then as in usual c#, you must reference it.&lt;/p&gt;
&lt;p&gt;When you run this cell, you should see the following output:&lt;/p&gt;
&lt;p&gt;&lt;img src="/posts/images/jupyter-notebook-csharp-r/sqlite01.png" class="img-fluid" alt="sqlite01" /&gt;&lt;/p&gt;
&lt;p&gt;We can then create a connection to an in-memory SQLite database.&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-CSharp"&gt;SQLiteConnection conn;

conn = new SQLiteConnection(&amp;quot;Data Source=:memory:;Version=3;New=True;&amp;quot;);

try
{
conn.Open();
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Creating two identical tables &lt;em&gt;SampleTable&lt;/em&gt; and &lt;em&gt;SampleTable1&lt;/em&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;using (SQLiteCommand sqlite_cmd = conn.CreateCommand()) {
string Createsql = &amp;quot;CREATE TABLE SampleTable(Col1 VARCHAR(20), Col2 INT)&amp;quot;;
string Createsql1 = &amp;quot;CREATE TABLE SampleTable1(Col1 VARCHAR(20), Col2 INT)&amp;quot;;
sqlite_cmd.CommandText = Createsql;
sqlite_cmd.ExecuteNonQuery();
sqlite_cmd.CommandText = Createsql1;
sqlite_cmd.ExecuteNonQuery();
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Inserting a set of data into these tables.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;using (SQLiteCommand sqlite_cmd = conn.CreateCommand()) {
sqlite_cmd.CommandText = &amp;quot;INSERT INTO SampleTable(Col1, Col2) VALUES ('Test Text ', 1);&amp;quot;;
sqlite_cmd.ExecuteNonQuery();
sqlite_cmd.CommandText = &amp;quot;INSERT INTO SampleTable(Col1, Col2) VALUES ('Test1 Text1 ', 2);&amp;quot;;
sqlite_cmd.ExecuteNonQuery();
sqlite_cmd.CommandText = &amp;quot;INSERT INTO SampleTable(Col1, Col2) VALUES ('Test2 Text2 ', 3);&amp;quot;;
sqlite_cmd.ExecuteNonQuery();
sqlite_cmd.CommandText = &amp;quot;INSERT INTO SampleTable1(Col1, Col2) VALUES ('Test3 Text3 ', 3);&amp;quot;;
sqlite_cmd.ExecuteNonQuery();
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Reading from &lt;em&gt;SampleTable&lt;/em&gt; to verify the insertions went through correctly.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;using (SQLiteCommand sqlite_cmd = conn.CreateCommand()) {
sqlite_cmd.CommandText = &amp;quot;SELECT * FROM SampleTable&amp;quot;;

using(var sqlite_datareader = sqlite_cmd.ExecuteReader()){
while (sqlite_datareader.Read())
{
string myreader = sqlite_datareader.GetString(0);
Console.WriteLine(myreader);
}
}
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If you executed the whole workbook up to now, you should have the following output.&lt;/p&gt;
&lt;p&gt;&lt;img src="/posts/images/jupyter-notebook-csharp-r/sqlite05.png" class="img-fluid" alt="sqlite05" /&gt;&lt;/p&gt;
&lt;p&gt;Closing the connection to the databse.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;conn.Close();
&lt;/code&gt;&lt;/pre&gt;
</content>
<summary>&lt;p&gt;Now that we have &lt;a href="/posts/jupyter-notebook-csharp-r"&gt;Jupyter Notebooks with C# installed&lt;/a&gt;, using it as an environment to play with SQLite is very easy. &lt;a href="https://www.sqlite.org/index.html"&gt;SQLite&lt;/a&gt; is a relational database that is small in footprint and self-contained. It also has a great in-memory mode which is perfect for playing around in a Jupyter notebook.&lt;/p&gt;</summary>
</entry>
</feed>
Loading

0 comments on commit a4a9dcf

Please sign in to comment.