Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix:when a mysql **Query** is greater than 16m, it will be crashed. #187

Closed
wants to merge 26 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,10 @@ test/jquery*
test/ct_log_cache
logs/*
.#*
.tags*
.rebar
include/crypto_compat.hrl
_build
rebar3.crashdump
rebar.lock
.rebar3
6 changes: 5 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,12 @@ before_script:
- make

script: "make test"

otp_release:
- 17.0-rc1
- 19.1
- 19.0
- 18.0
- 17.5
- R16B03-1
- R16B
- R15B03
Expand Down
5 changes: 5 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
**If you run into trouble, please try the 'stable' branch.**

git checkout origin/stable
#### Apr 27 2017 hd | doc
* timestamp optimize

#### Aug 26 2014 hd | doc
* Adapted for R18

#### Jun 21 2013 hd | doc
* Remove utf-8 encoding of binary data
Expand Down
15 changes: 14 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,19 @@ APP_NAME=emysql
MODULES=$(shell ls -1 src/*.erl | awk -F[/.] '{ print $$2 }' | sed '$$q;s/$$/,/g')
MAKETIME=$(shell date)

## Check if we are on erlang version that has namespaced types
ERL_NT=$(shell escript ./support/ntype_check.escript)

## Check if we are on erlang version that has erlang:timestamp/0
ERL_TS=$(shell escript ./support/timestamp_check.escript)

ifeq ($(ERL_NT),true)
ERLC_NT_FLAG=-Dnamespaced_types
endif
ifeq ($(ERL_TS),true)
ERLC_TS_FLAG=-Dtimestamp_support
endif

all: crypto_compat app
(cd src;$(MAKE))

Expand Down Expand Up @@ -80,7 +93,7 @@ CT_RUN = ct_run \
CT_SUITES=environment basics conn_mgr

build-tests:
erlc -v -o test/ $(wildcard test/*.erl) -pa ebin/
erlc -v $(ERLC_NT_FLAG) $(ERLC_TS_FLAG) -o test/ $(wildcard test/*.erl) -pa ebin/

test: all build-tests
@mkdir -p logs
Expand Down
75 changes: 6 additions & 69 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,79 +1,16 @@
# Emysql

[![Build Status](https://travis-ci.org/Eonblast/Emysql.png?branch=master)](https://travis-ci.org/Eonblast/Emysql)
[![Build Status](https://travis-ci.org/qingchuwudi/Emysql.svg?branch=master)](https://travis-ci.org/qingchuwudi/Emysql)

To use this driver, see the [Samples] further down in this README document.

<hr/>

Emysql implements a stable driver toward the MySQL database. It currently support fairly recent versions (somewhere around 5.3+) and it is considered fairly stable in production.

The driver has several technical shortcomings:

* No clear protocol / connection pool separation
* No clear protocol / socket separation
* A very complicated connection pool management
* Uses the textual protocol in a lot of places where it shouldthe binary protocol
* The API could be better

*However,* this is probably the best MySQL driver out there for Erlang. The `erlang-mysql-driver` uses a problematic connection pool design for many use cases and is not suitable for general purpose use. This driver is.

# Versioning

Semantic versioning is used. Versions are given as `x.y.z`:

* `x` changes break backward compatibility
* `y` changes add new functionality—but does not break compatibility backwards
* `z` changes recognizes a new version, but is not supposed to break anything

# Changelog of recent changes

## Emysql 0.4.1

Spring cleaning in the repository:
This is from [Eonblast/Emysql](https://github.com/Eonblast/Emysql)

* Removed a lot of modules which is not part of the official API from the documentation.
* Deprecated the use of `emysql_util` for `emysql`.
* Made tests actually do something with the `emysql_util` calls.
* Moved function calls from `emysql_util` into the modules where they belong.

Change should be backwards compatible.

## Emysql 0.4.0

Introduced changes by Garrett Smith:

* Support connection timeouts in gen_tcp:connect/3. This allows the driver to better handle connection timeouts upon initial connections.
* Introduce `emysql:add_pool/2` as a proplist-based way of initializing a pool. This is going to be the preferred way of initializing pools in the future.

## Emysql 0.3.2

* We should be on Travis CI now with full tests running
* Pulled in lots of smaller changes which should be rather cosmetic fixes.
* R16 warnings eliminated (Jesse Gumm)
* Tried to consolidate simple parts of the driver

## Emysql 0.3.0

*Note:* Automatic conversion to the encoding of the underlying MySQL
server was removed in the 0.3.x branch. If you specify, e.g., utf-8
encoding, then the MySQL server will reject wrongly-encoded strings
and data, but the *driver* will not perform any encoding by itself
anymore.

It is now the driver *callers* responsibility to ensure that data is
properly encoded. This change makes it possible to pass binary BLOB
data to the MySQL server once again.

# Overview:

This is an Erlang MySQL driver, based on a rewrite at Electronic Arts. [Easy][Samples] to use, strong [connection pooling][Adding_a_Pool], [prepared statements][Executing_Prepared_Statements] & [stored procedures][Executing_Stored_Procedures]. Optimized for a central node architecture and OLTP.
To use this driver, see the [Samples] further down in this README document.

While you can use mysql via ODBC, you should see better performance when using a *driver* like Emysql. For [samples][Samples] and [docs][] see below. Read the brief on [choosing][Choosing] a package and about the [history][History] of the various MySQL drivers.
## Introduction

[Emysql][1] is a cleaner rewrite of [erlang-mysql-driver][2], see [History][]. This fork is a direct continuation of the original [emysql][1] with [fixes][], [updates][], more [documentation][docs] and [samples][Samples].
[Eonblast/Emysql/README](https://github.com/Eonblast/Emysql/README.md)

**This is the master branch. Should you run into problems, please report them by opening an issue at github and try if they go away by checking out the 'stable' branch. Thank you.**
[inaka/Emysql/README](https://github.com/inaka/Emysql/README.md)

<hr/>

Expand Down
1 change: 1 addition & 0 deletions doc/emsql.gliffy

Large diffs are not rendered by default.

57 changes: 34 additions & 23 deletions include/emysql.hrl
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@
%% Jacob Vorreuter <[email protected]>,
%% Henning Diedrich <[email protected]>,
%% Eonblast Corporation <http://www.eonblast.com>
%%
%%
%% Permission is hereby granted, free of charge, to any person
%% obtaining a copy of this software and associated documentation
%% files (the "Software"),to deal in the Software without restric-
%% tion, including without limitation the rights to use, copy,
%% tion, including without limitation the rights to use, copy,
%% modify, merge, publish, distribute, sublicense, and/or sell
%% copies of the Software, and to permit persons to whom the
%% Software is furnished to do so, subject to the following
%% Software is furnished to do so, subject to the following
%% conditions:
%%
%%
%% The above copyright notice and this permission notice shall be
%% included in all copies or substantial portions of the Software.
%%
Expand All @@ -26,38 +26,40 @@
%% OTHER DEALINGS IN THE SOFTWARE.


-record(pool, {pool_id :: atom(),
size :: number(),
user :: string(),
password :: string(),
host :: string(),
port :: number(),
database :: string(),
encoding :: utf8 | latin1 | {utf8, utf8_unicode_ci} | {utf8, utf8_general_ci},
available=queue:new() :: queue(),
locked=gb_trees:empty() :: gb_tree(),
waiting=queue:new() :: queue(),
start_cmds=[] :: string(),
conn_test_period=0 :: number(),
connect_timeout=infinity :: number() | infinity,
warnings=false :: boolean()}).

-record(emysql_connection, {id :: string(),
pool_id :: atom(),
encoding :: atom(), % maybe could be latin1 | utf8 ?
max_allowed_packet :: non_neg_integer(), % max_allowed_packet on bytes.
socket :: inet:socket(),
version :: number(),
thread_id :: number(),
caps :: number(),
language :: number,
prepared=gb_trees:empty(),
language :: number(),
prepared=gb_trees:empty(),
locked_at :: number(),
alive=true :: boolean(),
test_period=0 :: number(),
last_test_time=0 :: number(),
monitor_ref :: reference(),
warnings=false :: boolean()}).

-record(pool, {pool_id :: atom(),
size :: number(),
user :: string(),
password :: string(),
host :: string(),
port :: number(),
database :: string(),
encoding :: utf8 | latin1 | {utf8, utf8_unicode_ci} | {utf8, utf8_general_ci},
max_allowed_packet :: non_neg_integer(), % max_allowed_packet on bytes.
available=queue:new() :: emysql:t_queue(),
locked=gb_trees:empty() :: emysql:t_gb_tree(),
waiting=queue:new() :: emysql:t_queue(),
start_cmds=[] :: string(),
conn_test_period=0 :: number(),
connect_timeout=infinity :: number() | infinity,
warnings=false :: boolean()}).

-record(greeting, {protocol_version :: number(),
server_version :: binary(),
thread_id :: number(),
Expand Down Expand Up @@ -109,7 +111,7 @@

-record(result_packet, {seq_num :: number(),
field_list :: list(),
rows, extra}).
rows :: any(), extra :: any()}).

-define(TIMEOUT, 8000).
-define(LOCK_TIMEOUT, 5000).
Expand Down Expand Up @@ -180,3 +182,12 @@
% number of result set columns.
-define(SERVER_STATUS_METADATA_CHANGED, 1024).

% https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_packets.html
% Sending More Than 16Mb
%
% If the payload is larger than or equal to 224-1 bytes the length is set to 224-1 (ff ff ff) and a additional packets are sent with the rest of the payload until the payload of a packet is less than 224-1 bytes.
% Sending a payload of 16 777 215 (224-1) bytes looks like:
%
% ff ff ff 00 ...
% 00 00 00 01
-define(MAX_LENGTH_PAYLOAD, 16#ffffff).
2 changes: 2 additions & 0 deletions rebar.config
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
% -*- Erlang -*-
% vim: ts=4 sw=4 et ft=erlang
{erl_opts, [
{platform_define, "^([1-9][0-9][0-9].*?)|([2-9][0-9].*?)|(1[8-9])", namespaced_types},
nowarn_deprecated_type
]}.
{pre_hooks,[

{"linux|bsd|darwin|solaris", compile, "escript ./support/crypto_compat.escript"},
{"win32", compile, "escript.exe support/crypto_compat.escript"}
]}.
14 changes: 14 additions & 0 deletions rebar.config.script
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{exports, ExportList} = lists:keyfind(exports,1,erlang:module_info()),
Check = lists:member({timestamp,0},ExportList),
case Check of
true ->
case lists:keyfind(erl_opts, 1, CONFIG) of
false ->
CONFIG ++ [{erl_opts,[{d,timestamp_support}]}];
{erl_opts, Opts} ->
NewOpts = {erl_opts, Opts ++ [{d,timestamp_support}]},
lists:keyreplace(erl_opts, 1, CONFIG, NewOpts)
end;
false ->
CONFIG
end.
2 changes: 1 addition & 1 deletion src/emysql.app.src
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

{application, emysql, [
{description, "Emysql - Erlang MySQL driver"},
{vsn, "0.4.1"},
{vsn, "0.4.5"},
{modules, []},
{mod, {emysql_app, ["%MAKETIME%"]}},
{registered, [emysql_conn_mgr, emysql_sup]},
Expand Down
Loading