Porting as a strategy for learning new languages

dalecosp

I recently tried porting an admin script from PHP to Python as a prod to learn the language.

Anyone else tried that? Would you recommend that as a strategy for lang-learning?

Weedpacket

I do a fair bit of language->language porting.

I'd expect that foo->PHP would be a more profitable direction for learning foo (assuming familiarity with PHP); otherwise what you end up learning is how to write PHP programs using foo's syntax (instead of getting to grips with idiomatic foo).

NogDog

Sort of like Weedpacket stated, I'd be careful about converting directly from language A to language B. I might go through code A in order to develop a comprehensive set of functional requirements, which you could then use to engineer a solution in language B using its specific idioms, features, and tools. Of course, that's probably somewhat idealistic: schedule pressures may come into play in some instances. 🙂

dalecosp

Hmm; thanks for the wisdom.

I've been "meaning to read" up on Python for quite some time. Also have a script that's not scaling well. It was my hope that the Python version would run faster (and it did, but only about 5%), so I think the bottleneck must be the DB queries.

I did find it enlightening and confidence building; I will want to look closely and see if I can see evidence of what you're warning about.

NogDog wrote:
schedule pressures

Oh, there's plenty of that. One-armed paper hanger mode is fully ingrained here. Part of the reason I picked this particular project was that I could (at least partially) justify doing it @work (it would've been a better justification if I'd gotten more than a 5% speed-up, but I wasn't at all certain I could believe that Python would be that much faster than PHP, and, as I noted, I don't think the language is the bottleneck, per se; less RAM used by Python also, but that's also not a Big Deal(tm) ... it's all about server load, which is mostly about MySQL, MyISAM tables, count(*), etc.)

Thank you, gentlemen,

🙂

NogDog

Time for some EXPLAINing. 🙂

dalecosp

NogDog;11044809 wrote:
Time for some EXPLAINing. 🙂

I wish that would help; it's a bunch of COUNT() statements on MyISAM tables, required to meet the CEO's expectations.

NogDog

dalecosp;11044813 wrote:
I wish that would help; it's a bunch of COUNT() statements on MyISAM tables, required to meet the CEO's expectations.

And you already know that the relevant queries and tables/indexes are currently optimized as much as they can be?

Derokorian

COUNT9) with a where clause of any kind, or a join can indeed require EXPLAINing. To make sure the join is using indexes and the where clause as well.

sneakyimp

If you want to know where your program is bogging down, I'd recommend using APD in PHP. For python, my friend wrote something called log_calls for profiling a Python script.

I would agree that EXPLAIN can provide extremely useful revelations about why a query might be slow. Good old Sxooter, Roger Ramjet, and a few others enlightened me a great deal about a particular performance problem I was tackling and I learned a great deal using EXPLAIN and partitioning as well. Distilling that massive thread down to its essence, I learned that partitioning tables can help solve specific performance problems -- especially if you partition across multiple physical storage devices when the db server is I/O-bound.

I think the key to learning a new language well is to expose yourself to especially well-written examples written in that language. Googling "best examples of python code" yielded this. Unfortunately the projects they list as good examples don't look like anything I need to work with.

dalecosp

As I said, it was "select count()" on MyISAM; this is well-documented as being dog-slow in MySQL.

Something interesting I did learn while researching was that the MySQL query cache is case-sensitive (and byte-sensitive, too), so, assuming none of the following have been issued within a set time frame, NONE of them will be served from the query_cache:

select foo from bar where baz=123;
select foo from bar where baz = 123;
select foo from bar WHERE baz=123;
SELECT foo from bar where baz=123;

and so on.

I spent some time looking through the "commercial" code one of our sites was built on; they apparent didn't know either as there were UPPER and lower-case SQL statements within the same file and even sometimes in the same method.

Chalk one up for having a rigid coding-style manual 😉

The current solution, which I dreamed up just prior to a meeting with an "outside" expert who suggested the same thing (WHEW! Glad I included that as a bullet-point in the meeting agenda!), was to have another box analyze the DB and write a summary table of counts back to the live box.

sneakyimp

dalecosp;11045039 wrote:
Something interesting I did learn while researching was that the MySQL query cache is case-sensitive (and byte-sensitive, too), so, assuming none of the following have been issued within a set time frame, NONE of them will be served from the query_cache:
select foo from bar where baz=123;
select foo from bar where baz = 123;
select foo from bar WHERE baz=123;
SELECT foo from bar where baz=123;
and so on.

What?? That seems like the silly sort of shortcut I might pull. I.e., just md5 the query string to create a unique id for it. A more thorough analysis should surely be done to improve the behavior of the query cache.

dalecosp

sneakyimp;11045049 wrote:
What?? That seems like the silly sort of shortcut I might pull. I.e., just md5 the query string to create a unique id for it. A more thorough analysis should surely be done to improve the behavior of the query cache.

I was a tad surprised, myself:

http://dev.mysql.com/doc/refman/5.7/en/query-cache-operation.html wrote:
Incoming queries are compared to those in the query cache before parsing, so the following two queries are regarded as different by the query cache:

SELECT FROM tbl_name
Select from tbl_name

Queries must be exactly the same (byte for byte) to be seen as identical. In addition, query strings that are identical may be treated as different for other reasons. Queries that use different databases, different protocol versions, or different default character sets are considered different queries and are cached separately.

Weedpacket

NogDog;11044809 wrote:
Time for some EXPLAINing. 🙂

For some reason I read this as: "Yew got some 'splainin' to do, bo-oy!"

NogDog

Weedpacket;11045057 wrote:
For some reason I read this as: "Yew got some 'splainin' to do, bo-oy!"

It was, in fact, posted with that sort of overloaded meaning being intended. (Not for any particular reason other than that I could, mind you.)

dalecosp

Given that I've replied twice since then, with "count(words) from posts where user='dalecosp' and verbose='very'" returning TRUE, I think I've 'splained enough.

Right, Dad? :p

Oh, and server load has gone way down and pages served/minute are up about sixty percent ...

Weedpacket

No intention to suggest otherwise; I missed the rest of my post. Viz:

dalecosp;11045055 wrote:
I was a tad surprised, myself:

What sort of half-assed cache is that? Couldn't they even cache the parsed query (tree)?

johanafm

Weedpacket;11045077 wrote:
What sort of half-assed cache is that? Couldn't they even cache the parsed query (tree)?

I have no idea what they do, nor what possible solutions exist. But at a cursory and naive glance, I'd guess that they do cache the parsed query. But in order to realize that a second query results in the same tree, they first have to parse the second query as well.

dalecosp;11045039 wrote:

select foo from bar where baz=123;
select foo from bar where baz = 123;
...

I believe you will also need to use prepared statements to make full use of the cache for cases like

-- from...
select foo from bar where baz=123;
select foo from bar where baz=456;

-- to...
select foo from bar where baz=:number;

Weedpacket

johanafm wrote:
I have no idea what they do, nor what possible solutions exist. But at a cursory and naive glance, I'd guess that they do cache the parsed query. But in order to realize that a second query results in the same tree, they first have to parse the second query as well.

No, what they're claiming to cache is the result set.
The assumption is that a result set that is worth caching is worth caching because it's repeatedly being requested by the same query (which probably is going to be byte-for-byte identical each time...).
You don't need a full parse; just the tokenising step would be enough (a SELECT token is a SELECT token whether it was generated by scanning "SELECT" or "Select"). If you want a hash key from that then a canonical SQL text string could be regenerated.

dalecosp

Weedpacket;11045083 wrote:
No, what they're claiming to cache is the result set.
The assumption is that a result set that is worth caching is worth caching because it's repeatedly being requested by the same query (which probably is going to be byte-for-byte identical each time...).

Good chance of that being true, but general queries on important tables might come from one of a few different places; the site itself, the backend "control panel", RSS and email generators and certain cronjobs all come to mind. Not to mention you might run the query from the top loop, the middle loop, the last loop in a script, and, as I noted, some of the previous hands in our code didn't bother checking about this.

NogDog

Time for a Solr collection? 🙂 (We use it on some of our apps for certain searches that are (a) done a lot and (b) not necessarily really quick.)